pytest.raises makes it impossible to check error messages and it should NOT accept parent classes by default

[pytestbot]

Originally reported by: Peter Hillerström (BitBucket: peterhil, GitHub: peterhil)

---

Pytest.raises code seems to be too complicated.

Find attached a code for simple 'raises' -context manager and
three simple tests for both. See the failures in the test run at the end.

#!python

#!/usr/bin/env python
# encoding: utf-8

import pytest
from contextlib import contextmanager


class StrictTypeError(TypeError):
    """
    Strict type error.
    """
    def __init__(self, message):
        self.message = message

    def __str__(self):
        return str(self.message)

    def __repr__(self):
        return "{0}({1})".format(self.__class__.__name__, repr(self.message))


@contextmanager
def raises(ExpectedException, message=''):
    try:
        yield
        raise AssertionError("Did not raise expected exception {0}!".format(repr(ExpectedException.__name__)))
    except ExpectedException, err:
        if message:
            assert message in err.message, 'expected message "{0}" not in "{1}"'.format(message, err.message)
        pass
    except Exception, err:
        raise AssertionError("Expected exception {0}, not\n\t{1}.".format(repr(ExpectedException.__name__), repr(err)))


class TestRaises(object):

    def test_raises(self):
        with raises(StrictTypeError, 'this should PASS'):
            raise StrictTypeError('this should PASS')

    def test_raises_message(self):
        with raises(AssertionError, 'expected message'):

            with raises(StrictTypeError, "this should FAIL because of the messages don't match"):
                raise StrictTypeError("not callable.")

            pytest.fail("Did not FAIL!")

    def test_raises_wrong_exception(self):
        try:

            with raises(StrictTypeError, 'Wrong exception class.'):
                raise TypeError('Wrong exception class.')

        except AssertionError, err:
            assert "Expected exception 'StrictTypeError', not\n\tTypeError" in err.message, "Failed with wrong reasons."
            pass
        except Exception, err:
            pytest.fail('Wrong exception!')


class TestPytestRaises(object):

    def test_pytest_raises(self):
        with pytest.raises(StrictTypeError) as err:
            assert 'this should PASS' in str(err)
            # assert 'this should PASS' in err.message
            raise StrictTypeError('this should PASS')

    def test_pytest_raises_message(self):
        with raises(AssertionError, 'expected message'):

            with pytest.raises(StrictTypeError) as err:
                try:
                    raise StrictTypeError('this should FAIL')
                except StrictTypeError, err:
                    assert isinstance(err, StrictTypeError)
                    assert issubclass(type(err), StrictTypeError)
                    raise err

            pytest.fail("Did not FAIL!")

    def test_pytest_raises_wrong_exception(self):
        try:

            with pytest.raises(StrictTypeError):
                raise TypeError("I'm a TypeError (wrong class).")

        except AssertionError, err:
            assert "Expected exception 'StrictTypeError', not\n\tTypeError" in err.message, "Failed with wrong reasons."
            pass
        except Exception, err:
            pytest.fail('Wrong exception! {0}'.format(err))

Example test run:

(py27)➜  sagitta-git git:(master) ✗ py.test-2.7 -v --assert=plain --tb=short sagitta/test/test_pytest_raises.py
============================================= test session starts =============================================
platform darwin -- Python 2.7.3 -- pytest-2.3.4 -- /Users/peterhil/Ohjelmointi/_Projects/_Functional/Sagitta/sagitta-git/venv/py27/bin/python
collected 6 items 

sagitta/test/test_pytest_raises.py:36: TestRaises.test_raises PASSED
sagitta/test/test_pytest_raises.py:40: TestRaises.test_raises_message PASSED
sagitta/test/test_pytest_raises.py:48: TestRaises.test_raises_wrong_exception PASSED
sagitta/test/test_pytest_raises.py:63: TestPytestRaises.test_pytest_raises FAILED
sagitta/test/test_pytest_raises.py:69: TestPytestRaises.test_pytest_raises_message FAILED
sagitta/test/test_pytest_raises.py:82: TestPytestRaises.test_pytest_raises_wrong_exception FAILED

================================================== FAILURES ===================================================
_____________________________________ TestPytestRaises.test_pytest_raises _____________________________________
sagitta/test/test_pytest_raises.py:65: in test_pytest_raises
>           assert 'this should PASS' in str(err)
venv/py27/lib/python2.7/site-packages/py/_code/code.py:395: in __str__
>       entry = self.traceback[-1]
E       AttributeError: 'ExceptionInfo' object has no attribute 'traceback'
_________________________________ TestPytestRaises.test_pytest_raises_message _________________________________
sagitta/test/test_pytest_raises.py:80: in test_pytest_raises_message
>           pytest.fail("Did not FAIL!")
/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/contextlib.py:35: in __exit__
>               self.gen.throw(type, value, traceback)
sagitta/test/test_pytest_raises.py:31: in raises
>           raise AssertionError("Expected exception {0}, not\n\t{1}.".format(repr(ExpectedException.__name__), repr(err)))
E           AssertionError: Expected exception 'AssertionError', not
E               Did not FAIL!.
_____________________________ TestPytestRaises.test_pytest_raises_wrong_exception _____________________________
sagitta/test/test_pytest_raises.py:92: in test_pytest_raises_wrong_exception
>           pytest.fail('Wrong exception! {0}'.format(err))
E           Failed: Wrong exception! I'm a TypeError (wrong class).
===================================== 3 failed, 3 passed in 0.12 seconds ======================================

---

• Bitbucket: https://bitbucket.org/pytest-dev/pytest/issue/258

[pytestbot]

Original comment by Ronny Pfannschmidt (BitBucket: RonnyPfannschmidt, GitHub: RonnyPfannschmidt):

---

your test is wrong

a pytest.fail raises a pytest.fail.Exception, not an assertionerror

[pytestbot]

Original comment by Ronny Pfannschmidt (BitBucket: RonnyPfannschmidt, GitHub: RonnyPfannschmidt):

---

and the other test is expecting time travel

[pytestbot]

Original comment by Ronny Pfannschmidt (BitBucket: RonnyPfannschmidt, GitHub: RonnyPfannschmidt):

---

all your tests on pytest.raises make wrong assumptions

[pytestbot]

Original comment by Ronny Pfannschmidt (BitBucket: RonnyPfannschmidt, GitHub: RonnyPfannschmidt):

---

the only problem i discovered is that pytest.fail.Exception has a empty string in the message attribute, the rest os pretty much correct

[pytestbot]

Original comment by Ronny Pfannschmidt (BitBucket: RonnyPfannschmidt, GitHub: RonnyPfannschmidt):

---

i just pushed a bugfix about exception attributes, other than that there is nothing wrong with pytest.raises

we might want to add a raises.regex that in addition to the exception takes a Regex
but that should be subject to a valid bugreport

[markfink]

I understand that you closed the issue but I fell into the same semantics trap. I believe this justifies a proper answer...

this is what I tried:

def test_pytest_raises_message():
    with pytest.raises(Exception, message='expected message'):
        raise Exception('this should FAIL')

    pytest.fail("Did not FAIL!")

this is how its done (notice the "match" part)

def test_pytest_raises_message():
    with pytest.raises(Exception, match='expected message'):
        raise Exception('this should FAIL')

    pytest.fail("Did not FAIL!")

[The-Compiler]

@markfink You probably want match='expected message'. From the documentation:

• message – if specified, provides a custom failure message if the exception is not raised
• match – if specified, asserts that the exception matches a text or regex

[RonnyPfannschmidt]

@The-Compiler i believe we should fail with unexpected parameters to raises

edit yikes i did misread, sorry

[nicoddemus]

Definitely, opened #3348 for that.