Number "lines to cover" Clover vs Jacoco on Sonarqube

[mgonzcast]

Our team has switched from Jacoco to Clover to ingest coverage reports into Sonarqube 8.9.2 Community Edition.

I have performed scans over two different projects with exactly the same code. One project I am using Clover and the project I am using Jacoco.

I have read in some forums that there might be a discrepancy on how the stastistics are calculated but I find weird that the overall "Lines to cover" under Jacoco are around 1k and under Clover are around 250 lines.

Checking the Measures tab under Sonarqube and running through the different packages I find that same java package reports 292 lines to cover in Jacoco based project while it reports only 62 lines to cover under the Clover project.

If I click on the top right "hamburguer" menu the "File measures" option. the number of lines, lines of code and comment lines are exactly the same but on the Coverage column is where the discrepancy shows.

Any idea? @marek-parfianowicz ? I also have raised an issue in the clover sonar plugin sfeir-open-source/sonar-clover#57

[marek-parfianowicz]

Hi! It's quite hard to tell without seeing two reports side-by-side. Would it be possible to see screen shots of source files with coverage highlighting for OpenClover and JaCoCo? Feel free to blur the code, if necessary. A single source file having the highest difference should be sufficient.

At the moment I can think about the following possibilities:

1. Test optimization in OpenClover. Your build ran with test optimization in OpenClover, as a result only a subset of tests was executed, leading to less lines covered in the code under test. Check if you do not use test optimization.

2. Code comments. The source file might have code comments. Does JaCoCo count lines with comments or not? As far as I remember, Clover does not. You can easily check this in the HTML report, lines with no executable code have no red/green marker.

3. Different interpretation of metrics. OpenClover primary metrics are: methods, statements and branches. A single statement can span over several source lines and vice versa, a single line might contain multiple statements. The line coverage metric in OpenClover is a derived metric. As far as I know, JaCoCo uses lines of code as primary metric. A careful comparison of reports from JaCoCo and OpenClover on a source code level could reveal this.

4. Some data transformation made in SonarQube. On the screen shots attached, I see 'Lines to cover' field. There is no such metric in OpenClover. So the question is: what data from clover.xml Clover SonarQube plugin is taking?

[marek-parfianowicz]

ad 4. Judging from the https://github.com/sfeir-open-source/sonar-clover/blob/master/src/main/java/org/sonar/plugins/clover/CloverXmlReportParser.java#L113, it skips the tag in the clover.xml file. Instead of this, it counts lines "manually", see https://github.com/sfeir-open-source/sonar-clover/blob/master/src/main/java/org/sonar/plugins/clover/CloverXmlReportParser.java#L162.

[mgonzcast]

Thanks @marek-parfianowicz for your quick answer!

From the Sonarqube forum https://community.sonarsource.com/t/number-lines-to-cover-clover-vs-jacoco/55832/5

they say "Lines to cover" means "Executable lines"

I am not sure we are talking about the same thing, maybe you mean lines of code?

As in my comment in the sonar clover plugin forum, I attach what I mean:

Clover: https://imgur.com/a/o6hnKDN

Jacoco: https://imgur.com/a/MAkNj68

[marek-parfianowicz]

As I wrote, I'd have to see full coverage information (either a Clover database with coverage recordings or HTML report; same for JaCoCo), ideally with build logs, in order to determine the root cause. You can contact me via [email protected].

[mgonzcast]

Thanks @marek-parfianowicz The issue here is how I anonymized the data in the report.

Can you tell me exactly which kind of report you mean with this phrase "Would it be possible to see screen shots of source files with coverage highlighting for OpenClover and JaCoCo? Feel free to blur the code, if necessary. A single source file having the highest difference should be sufficient" ?

If you attach a screenshot of what you mean maybe I can reproduce it and send it blurred as you say.

Also, again, I am not sure if we are talking about the same thing. From the Sonarqube forum:

https://community.sonarsource.com/t/number-lines-to-cover-clover-vs-jacoco/55832/5

They say "Lines to cover" means "Executable lines". I´m just wondering why Jacoco and Clover differ so much on that calculation. Also I wonder how meaningful is that deviation since overall coverage numbers are pretty similar.