Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Drop Codecov #12

Merged
merged 9 commits into from
Apr 12, 2022
Merged

Drop Codecov #12

merged 9 commits into from
Apr 12, 2022

Conversation

br3ndonland
Copy link
Owner

@br3ndonland br3ndonland commented Apr 11, 2022

Description

Codecov is a platform for displaying and tracking test coverage. Codecov's main value add is coverage delta (how a PR or commit will change coverage, compared with the previous result). Another minor value add is having the GitHub app comment on PRs.

However, Codecov doesn't actually measure test coverage. Other test coverage reporting tools, like coverage.py for Python projects, already do most of the work.

There are also some potential drawbacks to using Codecov:

  • Codecov had a security breach which necessitated changes to their tooling, and thereby necessitated additional updates and work for all their users.
  • Codecov is instituting report upload limits for private repos: "As of April 2nd, 2022 Codecov will only allow 250 coverage uploads to private/closed-source repositories per month on our free tier."
  • The Codecov upload action intermittently fails for no apparent reason (Error: failed to properly upload codecov/codecov-action#598).
  • The Codecov GitHub app intermittently stops sending status checks and PR comments.
  • The Codecov web app uses too many cookies. Why are 9 cookies needed to look at test coverage reports?
  • The Codecov web app has frequent OAuth issues with private repos in enterprise orgs. It prompts to "add private scope" repeatedly.
  • Codecov flags sometimes incorrectly report coverage. For the integration tests in this repo, Codecov says the report covers 100%, when coverage.py says it only covers 55% (the remainder being covered by unit tests).
  • Codecov YAML configuration can be confusing.

For this project, the drawbacks of using Codecov outweigh the benefits, so Codecov will no longer be used. Test coverage reporting will be performed with coverage.py.

Changes

  • Remove Codecov YAML (a834fac)
  • Remove Codecov badge from README (e832aa5)
  • Add shields.io badge for test coverage (df03d14)
  • Remove Codecov upload from GitHub Actions workflow (8b464e6)
  • Remove pytest-cov and just use coverage.py (fd29a0f)
  • Raise coverage of tests to 100% (82a7866)
  • Run pytest with coverage.py in GitHub Actions (962458c)
  • Fix inadvertent pytest fixture session token usage (5532293)
  • Specify source files for coverage.py (fca4234)

Related

@vercel
Copy link

vercel bot commented Apr 11, 2022

This pull request is being automatically deployed with Vercel (learn more).
To see the status of your deployment, click below or on the icon next to each commit.

🔍 Inspect: https://vercel.com/br3ndonland/fastenv/8pxpepQvUNjBR21AjmNmPeXfx9pE
✅ Preview: https://fastenv-git-drop-codecov-br3ndonland.vercel.app

pytest-cov was previously just measuring coverage of the source code.
This commit will increase coverage so the tests can be included also.

- Remove unused pytest fixture
- add `# pragma: no cover` comments to `pytest.skip` statements
- Update `tests/cloud/test_object_storage.py` to bring coverage to 100%
This commit will correct the two failing tests:

```text
FAILED tests/cloud/test_object_storage.py::TestObjectStorageClientUnit::
test_generate_presigned_url_example[False]

FAILED tests/cloud/test_object_storage.py::TestObjectStorageClientUnit::
test_generate_presigned_post_example[False]
```

The `False` parameter output means that `use_session_token` was `False`
in the corresponding fixtures in conftest.py:

- `object_storage_config_for_presigned_url_example`
- `object_storage_config_for_presigned_post_example`

There may be situations, such as tests in a GitHub Actions workflow, in
which multiple sets of credentials are present. In these situations,
there may be environment variables such as `AWS_SESSION_TOKEN` that are
set, but should not be used in certain `ObjectStorageConfig` instances.
Setting `session_token` to an empty string (`session_token=""`) prevents
`class ObjectStorageConfig` from using the environment variable value.

There is a unit test, `test_config_with_environment_variable_overrides`
in `tests/cloud/test_object_storage.py`, which verifies the correct
behavior. However, the pytest fixtures listed above were still setting
`session_token=None`, which then was making their `ObjectStorageConfig`
instances inadvertently pick up `AWS_SESSION_TOKEN`. The fix is just to
set `session_token=""` when `use_session_token=False` in those fixtures.
@sourcery-ai
Copy link
Contributor

sourcery-ai bot commented Apr 12, 2022

Sourcery Code Quality Report

✅  Merging this PR will increase code quality in the affected files by 0.17%.

Quality metrics Before After Change
Complexity 1.93 ⭐ 1.90 ⭐ -0.03 👍
Method Length 67.44 🙂 67.46 🙂 0.02 👎
Working memory 8.27 🙂 8.18 🙂 -0.09 👍
Quality 71.57% 🙂 71.74% 🙂 0.17% 👍
Other metrics Before After Change
Lines 1675 1643 -32
Changed files Quality Before Quality After Quality Change
tests/conftest.py 84.43% ⭐ 84.63% ⭐ 0.20% 👍
tests/cloud/test_object_storage.py 66.77% 🙂 67.13% 🙂 0.36% 👍

Here are some functions in these files that still need a tune-up:

File Function Complexity Length Working Memory Quality Recommendation
tests/cloud/test_object_storage.py TestObjectStorageClientUnit.test_create_presigned_post_policy 12 🙂 320 ⛔ 13 😞 35.90% 😞 Try splitting into smaller methods. Extract out complex expressions
tests/cloud/test_object_storage.py TestObjectStorageClientUnit.test_create_canonical_request_for_presigned_url_example 3 ⭐ 160 😞 11 😞 57.01% 🙂 Try splitting into smaller methods. Extract out complex expressions
tests/cloud/test_object_storage.py TestObjectStorageClientUnit.test_generate_presigned_post_example 2 ⭐ 142 😞 12 😞 58.16% 🙂 Try splitting into smaller methods. Extract out complex expressions
tests/cloud/test_object_storage.py TestObjectStorageClientIntegration.test_download_to_file_with_object_storage_config 1 ⭐ 155 😞 11 😞 59.50% 🙂 Try splitting into smaller methods. Extract out complex expressions
tests/cloud/test_object_storage.py TestObjectStorageClientIntegration.test_download_to_string_with_object_storage_config 1 ⭐ 129 😞 10 😞 64.28% 🙂 Try splitting into smaller methods. Extract out complex expressions

Legend and Explanation

The emojis denote the absolute quality of the code:

  • ⭐ excellent
  • 🙂 good
  • 😞 poor
  • ⛔ very poor

The 👍 and 👎 indicate whether the quality has improved or gotten worse with this pull request.


Please see our documentation here for details on how these metrics are calculated.

We are actively working on this report - lots more documentation and extra metrics to come!

Help us improve this quality report!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
1 participant