Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Address tests that are flaky on GHA #712

Merged
merged 12 commits into from
May 21, 2023
Merged

Address tests that are flaky on GHA #712

merged 12 commits into from
May 21, 2023

Conversation

khaeru
Copy link
Member

@khaeru khaeru commented May 20, 2023

Some of the tutorial tests are "flaky": they occasionally fail on the GitHub Actions runners, e.g. here. This necessitates daily work to restart the CI runs.

Per the Pytest docs on flaky tests, this PR:

  • Adds pytest-rerunfailures to the tests dependencies.
  • Applies the "flaky" marker to these tutorial tests.

Also:

  • Advertise Python 3.11 support in classifiers, and test against Python 3.11 on CI (parallel to Support Python 3.11 ixmp#481).
  • Update deprecated usage in the test suite: mainly of float(…) to get a single value from a pandas.DataFrame; using DataFrame.at[…] or .item() instead.

How to review

Confirm at least one instance where a test initially fails, and the usage of the plugin succeeds in allowing the test to pass on re-run.

This should appear somewhat like this in the log output:

test_all.py::test_example RERUN   [100%]
test_all.py::test_example RERUN   [100%]
test_all.py::test_example PASSED  [100%]

====== 1 passed, 2 rerun in 0.01s ======

→ here, at L314, re-run at L318, and summarized at L17008.

PR checklist

  • Continuous integration checks all ✅
  • Add or expand tests; coverage checks both ✅
  • Add, expand, or update documentation. N/A, testing only
  • Update release notes.

@khaeru khaeru added enh New features & functionality ci Continuous integration labels May 20, 2023
@khaeru khaeru self-assigned this May 20, 2023
@codecov
Copy link

codecov bot commented May 20, 2023

Codecov Report

Merging #712 (9ef7a04) into main (6bfc29e) will decrease coverage by 0.1%.
The diff coverage is 100.0%.

@@           Coverage Diff           @@
##            main    #712     +/-   ##
=======================================
- Coverage   94.4%   94.4%   -0.1%     
=======================================
  Files         43      43             
  Lines       3449    3448      -1     
=======================================
- Hits        3258    3257      -1     
  Misses       191     191             
Impacted Files Coverage Δ
...age_ix/tests/test_feature_bound_activity_shares.py 100.0% <ø> (ø)
message_ix/tests/test_core.py 100.0% <100.0%> (ø)
message_ix/tests/test_feature_bound_emission.py 100.0% <100.0%> (ø)
message_ix/tests/test_feature_capacity_factor.py 100.0% <100.0%> (ø)
message_ix/tests/test_feature_duration_time.py 100.0% <100.0%> (ø)
message_ix/tests/test_feature_storage.py 100.0% <100.0%> (ø)
message_ix/tests/test_feature_temporal_level.py 100.0% <100.0%> (ø)
message_ix/tests/test_soft_constraints.py 100.0% <100.0%> (ø)
message_ix/tests/test_tutorials.py 100.0% <100.0%> (ø)
message_ix/tests/tools/test_add_year.py 95.6% <100.0%> (-0.2%) ⬇️

@khaeru
Copy link
Member Author

khaeru commented May 21, 2023

We still see the following for macOS; all Python versions; for the R_austria tutorials:

Error in eval(expr, envir, enclos): java.lang.java.lang.UnsatisfiedLinkError: java.lang.UnsatisfiedLinkError: /Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/ixmp/backend/jdbc/libgmdjni64.dylib: dlopen(/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/ixmp/backend/jdbc/libgmdjni64.dylib, 0x0001): Library not loaded: '@rpath/libgcc_s.1.dylib'
<... omitted ...>n.framework/Versions/3.11/lib/python3.11/site-packages/ixmp/backend/jdbc/./libgcc_s.1.dylib' (no such file), '/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/ixmp/backend/jdbc/./libgcc_s.1.dylib' (no such file), '/Users/runner/hostedtoolcache/Java_Temurin-Hotspot_jdk/8.0.372-7/x64/Contents/Home/jre/lib/server/./libgcc_s.1.dylib' (no such file), '/Users/runner/hostedtoolcache/Java_Temurin-Hotspot_jdk/8.0.372-7/x64/Contents/Home/jre/lib/server/../libgcc_s.1.dylib' (no such file), '/Library/Frameworks/R.framework

@khaeru khaeru merged commit eb940c4 into main May 21, 2023
19 checks passed
@khaeru khaeru deleted the flaky branch May 21, 2023 14:28
@glatterf42
Copy link
Member

We still see the following for macOS; all Python versions; for the R_austria tutorials:

Error in eval(expr, envir, enclos): java.lang.java.lang.UnsatisfiedLinkError: java.lang.UnsatisfiedLinkError: /Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/ixmp/backend/jdbc/libgmdjni64.dylib: dlopen(/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/ixmp/backend/jdbc/libgmdjni64.dylib, 0x0001): Library not loaded: '@rpath/libgcc_s.1.dylib'
<... omitted ...>n.framework/Versions/3.11/lib/python3.11/site-packages/ixmp/backend/jdbc/./libgcc_s.1.dylib' (no such file), '/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/ixmp/backend/jdbc/./libgcc_s.1.dylib' (no such file), '/Users/runner/hostedtoolcache/Java_Temurin-Hotspot_jdk/8.0.372-7/x64/Contents/Home/jre/lib/server/./libgcc_s.1.dylib' (no such file), '/Users/runner/hostedtoolcache/Java_Temurin-Hotspot_jdk/8.0.372-7/x64/Contents/Home/jre/lib/server/../libgcc_s.1.dylib' (no such file), '/Library/Frameworks/R.framework

See iiasa/ixmp#479 for a related issue without solution as of now.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
ci Continuous integration enh New features & functionality
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants