-
-
Notifications
You must be signed in to change notification settings - Fork 428
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add branch details to JSON report #1438
Add branch details to JSON report #1438
Conversation
Added more branches to the code whose coverage is checked.
The json report now includes for each branch which branches have been executed, missed and what the percentage of covered branches was.
tests/test_json.py
Outdated
'excluded_lines': [], | ||
'branch_details': { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Mm, in Slipcover, I chose to output executed_branches
and missing_branches
directly next to executed_lines
, etc., as a list of tuples (which in JSON becomes a list of 2-element lists). With ->exit
branches being shown as [x,0]
"tuples" (where x
is the source line).
We don't have to have the same format, but it would be nice...
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Hey, thanks for the feedback. I changed the format to the one you requested.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Cool! Thank you.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'm curious why 0 is better than -1 for exit?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Mm, I think both work well. I just a hope Ed to pick 0. Do you have a reason why -1 is better?
.. Juan
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Internally, entering and exiting a code object is represented in arcs as negative numbers. From one of the docstrings:
Negative numbers have special meaning. If the starting line number is
-N, it represents an entry to the code object that starts at line N.
If the ending ling number is -N, it's an exit from the code object that
starts at line N.
Here's some actual data from the .coverage file:
.coverage> select * from arc;
+---------+------------+--------+------+
| file_id | context_id | fromno | tono |
+---------+------------+--------+------+
| 1 | 1 | 3 | 4 |
| 1 | 1 | 5 | -1 |
| 1 | 1 | 12 | 13 |
| 1 | 1 | 17 | 18 |
| 1 | 1 | 9 | 17 |
| 1 | 1 | 1 | 9 |
| 1 | 1 | 18 | -1 |
| 1 | 1 | 15 | 11 |
| 1 | 1 | -1 | 1 |
| 1 | 1 | 4 | 5 |
| 1 | 1 | 12 | 15 |
| 1 | 1 | 10 | 11 |
| 1 | 1 | -9 | 10 |
| 1 | 1 | 7 | 3 |
| 1 | 1 | 4 | 7 |
| 1 | 1 | 13 | -9 |
| 1 | 1 | 2 | 3 |
| 1 | 1 | 11 | 12 |
| 1 | 1 | -1 | 2 |
+---------+------------+--------+------+
I'm curious why you haven't seen negative numbers other than -1 in testing.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It's easy to get other negative numbers. This is using the code from this pull request:
# /tmp/b1438.py
def never(x):
print("ok")
if x: return 17
print("nope")
never(0)
then:
% coverage run --branch /tmp/b1438.py
ok
nope
% coverage report -m
Name Stmts Miss Branch BrPart Cover Missing
-------------------------------------------------------------------
/private/tmp/b1438.py 5 0 2 1 86% 5->exit
-------------------------------------------------------------------
TOTAL 5 0 2 1 86%
% coverage json
Wrote JSON report to coverage.json
% jq <coverage.json
{
"meta": {
"version": "6.4.5a0",
"timestamp": "2022-09-05T07:25:02.254430",
"branch_coverage": true,
"show_contexts": false
},
"files": {
"/private/tmp/b1438.py": {
"executed_lines": [
3,
4,
5,
6,
8
],
"summary": {
"covered_lines": 5,
"num_statements": 5,
"percent_covered": 85.71428571428571,
"percent_covered_display": "86",
"missing_lines": 0,
"excluded_lines": 0,
"num_branches": 2,
"num_partial_branches": 1,
"covered_branches": 1,
"missing_branches": 1
},
"missing_lines": [],
"excluded_lines": [],
"executed_branches": [
[
5,
6
]
],
"missing_branches": [
[
5,
-3
]
]
}
},
"totals": {
"covered_lines": 5,
"num_statements": 5,
"percent_covered": 85.71428571428571,
"percent_covered_display": "86",
"missing_lines": 0,
"excluded_lines": 0,
"num_branches": 2,
"num_partial_branches": 1,
"covered_branches": 1,
"missing_branches": 1
}
}
missing_branches mentions -3, because the exit was never taken from the code starting at line 3.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
For that .py
, Slipcover outputs a missing (5,5) branch, since really it is a same-line branch (to the return
statement), below. For branches that exit a function, it outputs 0 as the destination... As in the (2,0) branch for
def foo(x):
if x == 0:
x += 1
foo(0)
I could change that to -1, but to change that to something dependent upon where the code starts would take more effort.
{
"meta": {
"software": "slipcover",
"version": "0.2.0",
"timestamp": "2022-09-05T14:24:30.974682",
"branch_coverage": true
},
"files": {
"b1438.py": {
"executed_lines": [
3,
4,
5,
6,
8
],
"missing_lines": [],
"summary": {
"covered_lines": 5,
"missing_lines": 0,
"covered_branches": 1,
"missing_branches": 1,
"percent_covered": 85.71428571428571
},
"executed_branches": [
[
5,
6
]
],
"missing_branches": [
[
5,
5
]
]
}
},
"summary": {
"covered_lines": 5,
"missing_lines": 0,
"covered_branches": 1,
"missing_branches": 1,
"percent_covered": 85.71428571428571
}
}
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Maybe we need to talk about the value of coverage.py and slipcover producing the same data, vs coverage.py producing data with more information?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Right... Is there truly value in encoding the starting line of the code in the branch destination? How would that help a user?
I guess the more the packages differ in the data they collect, potentially the more difficult a Slipcover-based collector for coverage.py might become... which, of course, at this point is just an idea.
In this case in particular, it seems like we'd generate different data anyway, because of the same-line branch.
Executed and missing branch arcs are stored in the fields named 'executed_branches' and 'missing_branches' respectively. Both fields contain a list of two element lists. The first element represents the source line number and the second one the target line number. Exit branches have their target line number set to 0.
I've squashed and merged this as of 95195b1. I've kept the negative line numbers for entrances and exits, to keep the data uniform with the rest of coverage.py. Thanks! |
This is now released as part of coverage 6.5.0. |
Version 6.5.0 — 2022-09-29 -------------------------- - The JSON report now includes details of which branches were taken, and which are missing for each file. Thanks, Christoph Blessing (`pull 1438`_). Closes `issue 1425`_. - Starting with coverage.py 6.2, ``class`` statements were marked as a branch. This wasn't right, and has been reverted, fixing `issue 1449`_. Note this will very slightly reduce your coverage total if you are measuring branch coverage. - Packaging is now compliant with `PEP 517`_, closing `issue 1395`_. - A new debug option ``--debug=pathmap`` shows details of the remapping of paths that happens during combine due to the ``[paths]`` setting. - Fix an internal problem with caching of invalid Python parsing. Found by OSS-Fuzz, fixing their `bug 50381`_. .. _bug 50381: https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=50381 .. _PEP 517: https://peps.python.org/pep-0517/ .. _issue 1395: nedbat/coveragepy#1395 .. _issue 1425: nedbat/coveragepy#1425 .. _pull 1438: nedbat/coveragepy#1438 .. _issue 1449: nedbat/coveragepy#1449
[![Mend Renovate](https://app.renovatebot.com/images/banner.svg)](https://renovatebot.com) This PR contains the following updates: | Package | Change | Age | Adoption | Passing | Confidence | |---|---|---|---|---|---| | [coverage](https://togithub.com/nedbat/coveragepy) | `==6.4.2` -> `==7.1.0` | [![age](https://badges.renovateapi.com/packages/pypi/coverage/7.1.0/age-slim)](https://docs.renovatebot.com/merge-confidence/) | [![adoption](https://badges.renovateapi.com/packages/pypi/coverage/7.1.0/adoption-slim)](https://docs.renovatebot.com/merge-confidence/) | [![passing](https://badges.renovateapi.com/packages/pypi/coverage/7.1.0/compatibility-slim/6.4.2)](https://docs.renovatebot.com/merge-confidence/) | [![confidence](https://badges.renovateapi.com/packages/pypi/coverage/7.1.0/confidence-slim/6.4.2)](https://docs.renovatebot.com/merge-confidence/) | --- ### Release Notes <details> <summary>nedbat/coveragepy</summary> ### [`v7.1.0`](https://togithub.com/nedbat/coveragepy/blob/HEAD/CHANGES.rst#Version-710--2023-01-24) [Compare Source](https://togithub.com/nedbat/coveragepy/compare/7.0.5...7.1.0) - Added: the debug output file can now be specified with `[run] debug_file` in the configuration file. Closes `issue 1319`\_. - Performance: fixed a slowdown with dynamic contexts that's been around since 6.4.3. The fix closes `issue 1538`*. Thankfully this doesn't break the `Cython change`* that fixed `issue 972`\_. Thanks to Mathieu Kniewallner for the deep investigative work and comprehensive issue report. - Typing: all product and test code has type annotations. .. \_Cython change:[nedbat/coveragepy#1347 .. \_issue 972[nedbat/coveragepy#972 .. \_issue 131[nedbat/coveragepy#1319 .. \_issue 15[nedbat/coveragepy#1538 .. \_changes\_7-0-5: ### [`v7.0.5`](https://togithub.com/nedbat/coveragepy/blob/HEAD/CHANGES.rst#Version-705--2023-01-10) [Compare Source](https://togithub.com/nedbat/coveragepy/compare/7.0.4...7.0.5) - Fix: On Python 3.7, a file with type annotations but no `from __future__ import annotations` would be missing statements in the coverage report. This is now fixed, closing `issue 1524`\_. .. \_issue 1524:[nedbat/coveragepy#1524 .. \_changes\_7-0-4: ### [`v7.0.4`](https://togithub.com/nedbat/coveragepy/blob/HEAD/CHANGES.rst#Version-704--2023-01-07) [Compare Source](https://togithub.com/nedbat/coveragepy/compare/7.0.3...7.0.4) - Performance: an internal cache of file names was accidentally disabled, resulting in sometimes drastic reductions in performance. This is now fixed, closing `issue 1527`\_. Thanks to Ivan Ciuvalschii for the reproducible test case. .. \_issue 1527:[nedbat/coveragepy#1527 .. \_changes\_7-0-3: ### [`v7.0.3`](https://togithub.com/nedbat/coveragepy/blob/HEAD/CHANGES.rst#Version-703--2023-01-03) [Compare Source](https://togithub.com/nedbat/coveragepy/compare/7.0.2...7.0.3) - Fix: when using pytest-cov or pytest-xdist, or perhaps both, the combining step could fail with `assert row is not None` using 7.0.2. This was due to a race condition that has always been possible and is still possible. In 7.0.1 and before, the error was silently swallowed by the combining code. Now it will produce a message "Couldn't combine data file" and ignore the data file as it used to do before 7.0.2. Closes `issue 1522`\_. .. \_issue 1522:[nedbat/coveragepy#1522 .. \_changes\_7-0-2: ### [`v7.0.2`](https://togithub.com/nedbat/coveragepy/blob/HEAD/CHANGES.rst#Version-702--2023-01-02) [Compare Source](https://togithub.com/nedbat/coveragepy/compare/7.0.1...7.0.2) - Fix: when using the `[run] relative_files = True` setting, a relative `[paths]` pattern was still being made absolute. This is now fixed, closing `issue 1519`\_. - Fix: if Python doesn't provide tomllib, then TOML configuration files can only be read if coverage.py is installed with the `[toml]` extra. Coverage.py will raise an error if TOML support is not installed when it sees your settings are in a .toml file. But it didn't understand that `[tools.coverage]` was a valid section header, so the error wasn't reported if you used that header, and settings were silently ignored. This is now fixed, closing `issue 1516`\_. - Fix: adjusted how decorators are traced on PyPy 7.3.10, fixing `issue 1515`\_. - Fix: the `coverage lcov` report did not properly implement the `--fail-under=MIN` option. This has been fixed. - Refactor: added many type annotations, including a number of refactorings. This should not affect outward behavior, but they were a bit invasive in some places, so keep your eyes peeled for oddities. - Refactor: removed the vestigial and long untested support for Jython and IronPython. .. \_issue 1515:[nedbat/coveragepy#1515 .. \_issue 1516[nedbat/coveragepy#1516 .. \_issue 151[nedbat/coveragepy#1519 .. \_changes\_7-0-1: ### [`v7.0.1`](https://togithub.com/nedbat/coveragepy/blob/HEAD/CHANGES.rst#Version-701--2022-12-23) [Compare Source](https://togithub.com/nedbat/coveragepy/compare/7.0.0...7.0.1) - When checking if a file mapping resolved to a file that exists, we weren't considering files in .whl files. This is now fixed, closing `issue 1511`\_. - File pattern rules were too strict, forbidding plus signs and curly braces in directory and file names. This is now fixed, closing `issue 1513`\_. - Unusual Unicode or control characters in source files could prevent reporting. This is now fixed, closing `issue 1512`\_. - The PyPy wheel now installs on PyPy 3.7, 3.8, and 3.9, closing `issue 1510`\_. .. \_issue 1510:[nedbat/coveragepy#1510 .. \_issue 1511[nedbat/coveragepy#1511 .. \_issue 151[nedbat/coveragepy#1512 .. \_issue 15[nedbat/coveragepy#1513 .. \_changes\_7-0-0: ### [`v7.0.0`](https://togithub.com/nedbat/coveragepy/blob/HEAD/CHANGES.rst#Version-700--2022-12-18) [Compare Source](https://togithub.com/nedbat/coveragepy/compare/6.5.0...7.0.0) Nothing new beyond 7.0.0b1. .. \_changes\_7-0-0b1: ### [`v6.5.0`](https://togithub.com/nedbat/coveragepy/blob/HEAD/CHANGES.rst#Version-650--2022-09-29) [Compare Source](https://togithub.com/nedbat/coveragepy/compare/6.4.4...6.5.0) - The JSON report now includes details of which branches were taken, and which are missing for each file. Thanks, `Christoph Blessing <pull 1438_>`*. Closes `issue 1425`*. - Starting with coverage.py 6.2, `class` statements were marked as a branch. This wasn't right, and has been reverted, fixing `issue 1449`\_. Note this will very slightly reduce your coverage total if you are measuring branch coverage. - Packaging is now compliant with `PEP 517`*, closing `issue 1395`*. - A new debug option `--debug=pathmap` shows details of the remapping of paths that happens during combine due to the `[paths]` setting. - Fix an internal problem with caching of invalid Python parsing. Found by OSS-Fuzz, fixing their `bug 50381`\_. .. \_bug 50381: https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=50381 .. \_PEP 517: https://peps.python.org/pep-0517/ .. \_issue 139[nedbat/coveragepy#1395 .. \_issue 14[nedbat/coveragepy#1425 .. \_issue 1[nedbat/coveragepy#1449 .. \_pull [nedbat/coveragepy#1438 .. \_changes\_6-4-4: ### [`v6.4.4`](https://togithub.com/nedbat/coveragepy/blob/HEAD/CHANGES.rst#Version-644--2022-08-16) [Compare Source](https://togithub.com/nedbat/coveragepy/compare/6.4.3...6.4.4) - Wheels are now provided for Python 3.11. .. \_changes\_6-4-3: ### [`v6.4.3`](https://togithub.com/nedbat/coveragepy/blob/HEAD/CHANGES.rst#Version-643--2022-08-06) [Compare Source](https://togithub.com/nedbat/coveragepy/compare/6.4.2...6.4.3) - Fix a failure when combining data files if the file names contained glob-like patterns. Thanks, `Michael Krebs and Benjamin Schubert <pull 1405_>`\_. - Fix a messaging failure when combining Windows data files on a different drive than the current directory, closing `issue 1428`*. Thanks, `Lorenzo Micò <pull 1430_>`*. - Fix path calculations when running in the root directory, as you might do in a Docker container. Thanks `Arthur Rio <pull 1403_>`\_. - Filtering in the HTML report wouldn't work when reloading the index page. This is now fixed. Thanks, `Marc Legendre <pull 1413_>`\_. - Fix a problem with Cython code measurement, closing `issue 972`*. Thanks, `Matus Valo <pull 1347_>`*. .. \_issue 972:[nedbat/coveragepy#972 .. \_issue 1428[nedbat/coveragepy#1428 .. \_pull 134[nedbat/coveragepy#1347 .. \_pull 14[nedbat/coveragepy#1403 .. \_pull 1[nedbat/coveragepy#1405 .. \_pull [nedbat/coveragepy#1413 .. \_pull[nedbat/coveragepy#1430 .. \_changes\_6-4-2: </details> --- ### Configuration 📅 **Schedule**: Branch creation - At any time (no schedule defined), Automerge - At any time (no schedule defined). 🚦 **Automerge**: Disabled by config. Please merge this manually once you are satisfied. ♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox. 🔕 **Ignore**: Close this PR and you won't be reminded about this update again. --- - [ ] <!-- rebase-check -->If you want to rebase/retry this PR, check this box --- This PR has been generated by [Mend Renovate](https://www.mend.io/free-developer-tools/renovate/). View repository job log [here](https://app.renovatebot.com/dashboard#github/allenporter/flux-local). <!--renovate-debug:eyJjcmVhdGVkSW5WZXIiOiIzNC4xMjQuMiIsInVwZGF0ZWRJblZlciI6IjM0LjEyNC4yIn0=--> Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
Fixes: #1425