OC-29 JUnit @Rule ExpectedException marked as failure

[marek-parfianowicz]

In the Clover HTML output, Clover marks JUnit tests that use the JUnit @rule ExpectedException to check for customized exceptions as failures, even though the test actually passes. For example:

public void testGetSubscriptionScopeDetailNoCsScope() {
    expectedException.expect(YyyyException.class);
    expectedException.expect(ErrorMatcher.method(xxx));

    object.method();
}```

[marek-parfianowicz]

Hi Cesare, thank you for reporting this. This is a known limitation of Clover, see https://jira.atlassian.com/browse/CLOV-1063. Luckily there are two workarounds available, I would recommend to use a workaround no 2. See comments below.

[marek-parfianowicz]

Workaround 1
For Clover to understand that the test is passing you would need to add an 'expected' attribute to the Test annotation

public void testSomething() {

}```

[marek-parfianowicz]

Workaround 2
Configure Clover report task to use JUnit's XML files as a source of test results, instead of relying on Clover's internal tracking.

in Ant, use // tag, see https://confluence.atlassian.com/display/CLOVER/clover-report
in Maven, configure custom report configuration, see https://confluence.atlassian.com/display/CLOVER/Creating+custom+reports
in Gradle, use custom clover.reporttask closure, see https://confluence.atlassian.com/display/CLOVER/Advanced+report+configuration

[marek-parfianowicz]

Potential solutions of the problem
Extend Clover's grammar parser
Extend ANTLR grammar parser (java.g) and try to find Rule calls. Look up for 'expect(Class)' and 'expectMessage(String)' method calls. If such signature is found - mark the test method that it shall throw an exception.

This method would not be 100% accurate. I can either overlook some rules (for instance - simple moving rules to a helper method would break it) or report false positives (for instance - some calls a method called 'expect' not being a Rule).

Use Clover's runtime test monitoring feature
Clover 3.3 introduced support for JUnit4 and Spock parameterized tests (aka test iterations). Clover can monitor test execution at runtime and query test framework for some data (in this case - the real name of the test being executed). We could try to ask test framework for test results at runtime and use this information to pass or fail a test, instead of relying on catching exceptions (as it is done now).

[marek-parfianowicz]

Oliver Plaehn

Unfortunately, both workarounds have problems of their own.

Workaround 1 implies that you have to rewrite all tests that use "@ Rule ExpectedException" to use "@ Test(expected = Exception.class)" instead, because you can't use both. That is a lot of work and is an inferior approach because you don't have matchers for exception messages and causes.

Workaround 2 works but only impacts the report generation. During the "check" phase, tests using "@ Rule ExpectedException" are still considered failures and are thus not included in the coverage computation. This caused a lot of our modules to wrongly report a very low conditional coverage.

To work around the problem with workaround 2 (sorry, couldn't resist ;-), one can set the option "includeFailedTestCoverage" to "true" to include coverage from "failing" tests. That works for us because we consider a build to be a failure if a single tests fails, so we do not care about the case when failed tests artifically increase coverage rates.

Alas, the current version of the openclover maven plugin, which we use, does not expose this option in "pom.xml"s.

So we ended up patching the openclover maven plugin locally (easy enough to do).

To sum up: I can live with the fact that tests that use "@ Rule ExpectedException" are considered as failures. But please at least allow users of the maven plugin to use the "includeFailedTestCoverage" option.

I can provide a patch if you're interested but it is really only four lines of code.

[marek-parfianowicz]

Original report: https://bitbucket.org/openclover/clover/issues/29