Relevant line calculation difference between python versions, related to -0.0% change?

[jtroussard]

Description

Our pipeline runs coverallsapp action in parallel with several different versions of python. Noticed that after making a non-functional change to a file, (removed comment) our coveralls check was failing the build for a -0.0% decrease.

Difference Between Versions

Search results in a small difference the relevant lines are calculated between python versions:

3.7

3.8

The difference between the 2 are the method declaration lines. 3.6 and 3.7 seem to leave it out, while 3.8 and beyond include the line as "relevant". This is the only significant difference I can find between the passing coveralls checks (3.6, 3.7) and the failing checks, (3.8, 3.9)

coveralls github action results

Questions

• Can anyone shed any light on this issue?
• Is there a workaround besides increasing the failure threshold?
• Is this a bug?
• Am I misdiagnosing my coverage failures?

[afinetooth]

Hi @jtroussard.

Can anyone shed any light on this issue?

On the Coveralls side of things it's hard to account for differences between any two coverage reports received by the API. Our calculations are based on the data in each report, which is to say, we simply accept what the reports tell us and can't account for things like why the same coverage library determines one line to be relevant on one OS version and not relevant on the other.

The best you can do to ensure some reasonably expected uniformity in your original coverage reports is to use the same versions of your coverage libraries and generators across all jobs in a build (if possible).

If you want to share the Coveralls URL for your repo, and for any build(s) that appear to be affected, we're happy to look more deeply into the specific case. If your project is private, or sensitive, you can email us at [email protected] and refer to this issue.

Is this a bug?
Am I misdiagnosing my coverage failures?

There is a known issue affecting some projects whereby calculations that should result in a status of "no change" are being returned as decreases (of -0.0%).

I suspect you are affected and, if so, then it's not that you're misdiagnosing your coverage changes / failures, it's just that a particular difference between two parallel jobs in your build may or may not be relevant. It is possible that two builds could be one line number off, but that would typically result in a decrease that's more statistically significant than 0.0%.

Is there a workaround besides increasing the failure threshold?

I'm afraid not at this moment. If you're being affected by the known bug, then to avoid it you'll want to set a value larger than 0.0% for the setting: COVERAGE DECREASE THRESHOLD FOR FAILURE

You'll find that in your repo's settings page on coveralls, which is at:

https://coveralls.io/<your repo host>/<your org>/<your repo name>/settings

---

I have +1'd you on our internal card for the known issue. I'll post any updates here when we get them there.