Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Address tests that are flaky with pytest-xdist #512

Open
glatterf42 opened this issue Jan 15, 2024 · 2 comments · May be fixed by #521
Open

Address tests that are flaky with pytest-xdist #512

glatterf42 opened this issue Jan 15, 2024 · 2 comments · May be fixed by #521
Labels
bug ci Continuous integration
Milestone
3.10

Comments

@glatterf42
Copy link
Member

As mentioned in #510, pytest-xdist allows us to run tests on multiple tasks simultaneously, decreasing our run time. However, some tests show flaky behaviour since they attempt to read/write from processes they don't belong to. iiasa/message_ix#765 (comment) mentions a method with which to resolve this flakiness and successfully implements it on several tests in that repo. We should add the same/similar fixes here and then revert the mitigation added in #510.
@glatterf42 glatterf42 added bug ci Continuous integration labels Jan 15, 2024
@glatterf42 glatterf42 added this to the 3.9 milestone Jan 15, 2024
@glatterf42
Copy link
Member Author

Possibly related to #467.

@glatterf42
Copy link
Member Author

glatterf42 commented Mar 12, 2024 •
edited
Loading

Here's a list of flaky tests I recorded since 11.01.2024:

  • test_py_transport

    FAILED ixmp/tests/test_tutorials.py::test_py_transport - nbclient.exceptions.CellExecutionError: An error occurred while executing the following cell:
------------------
# launch the ix modeling platform using the default back end
mp = ixmp.Platform()

# The following lines have the same effect:
# mp = ixmp.Platform(name='local')  # Default local database
# mp = ixmp.Platform(name='default')  # Default database -> 'local'
------------------


---------------------------------------------------------------------------
Exception                                 Traceback (most recent call last)
File Platform.java:110, in at.ac.iiasa.ixmp.Platform.<init>()

File Platform.java:163, in at.ac.iiasa.ixmp.Platform.initPlatform()

File DbDAO.java:238, in at.ac.iiasa.ixmp.database.DbDAO.create()

File DbDAO.java:318, in at.ac.iiasa.ixmp.database.DbDAO.updateSchemaIfNecessary()

File Flyway.java:910, in org.flywaydb.core.Flyway.migrate()

File Flyway.java:1238, in org.flywaydb.core.Flyway.execute()

File Flyway.java:910, in org.flywaydb.core.Flyway$1.execute()

File Flyway.java:918, in org.flywaydb.core.Flyway$1.execute()

File Flyway.java:78, in org.flywaydb.core.Flyway.access$100()

File Flyway.java:1037, in org.flywaydb.core.Flyway.doValidate()

Exception: Java Exception

The above exception was the direct cause of the following exception:

org.flywaydb.core.api.FlywayException     Traceback (most recent call last)
File ~/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/ixmp/backend/jdbc.py:297, in JDBCBackend.__init__(self, jvmargs, **kwargs)
    296 try:
--> 297     self.jobj = java.Platform("Python", properties)
    298 except java.NoClassDefFoundError as e:  # pragma: no cover

org.flywaydb.core.api.FlywayException: org.flywaydb.core.api.FlywayException: Validate failed: Detected failed migration to version 1 (hsql base version)

During handling of the above exception, another exception occurred:

RuntimeError                              Traceback (most recent call last)
Cell In[2], line 2
      1 # launch the ix modeling platform using the default back end
----> 2 mp = ixmp.Platform()
      4 # The following lines have the same effect:
      5 # mp = ixmp.Platform(name='local')  # Default local database
      6 # mp = ixmp.Platform(name='default')  # Default database -> 'local'

File ~/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/ixmp/core/platform.py:94, in Platform.__init__(self, name, backend, **backend_args)
     88     raise ValueError(
     89         f"backend class {repr(backend_class_name)} not among "
     90         + str(sorted(BACKENDS.keys()))
     91     )
     93 # Instantiate the backend
---> 94 self._backend = backend_class(**kwargs)

File ~/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/ixmp/backend/jdbc.py:320, in JDBCBackend.__init__(self, jvmargs, **kwargs)
    318     else:
    319         _raise_jexception(e)
--> 320     raise RuntimeError(f"{msg}\n(Java: {jclass})")
    322 # Set the log level
    323 self.set_log_level(log_level)

RuntimeError: when initializing database:
(Java: org.flywaydb.core.api.FlywayException)
On: macos-latest-py3.8, macos-latest-py3.10

  • test_py_transport_scenario

    FAILED ixmp/tests/test_tutorials.py::test_py_transport_scenario - nbclient.exceptions.CellExecutionError: An error occurred while executing the following cell:
------------------
# launch the ix modeling platform using the default local database
mp = ixmp.Platform()
------------------


---------------------------------------------------------------------------
Exception                                 Traceback (most recent call last)
File Platform.java:110, in at.ac.iiasa.ixmp.Platform.<init>()

File Platform.java:163, in at.ac.iiasa.ixmp.Platform.initPlatform()

File DbDAO.java:238, in at.ac.iiasa.ixmp.database.DbDAO.create()

File DbDAO.java:318, in at.ac.iiasa.ixmp.database.DbDAO.updateSchemaIfNecessary()

File Flyway.java:910, in org.flywaydb.core.Flyway.migrate()

File Flyway.java:1238, in org.flywaydb.core.Flyway.execute()

File Flyway.java:910, in org.flywaydb.core.Flyway$1.execute()

File Flyway.java:918, in org.flywaydb.core.Flyway$1.execute()

File Flyway.java:78, in org.flywaydb.core.Flyway.access$100()

File Flyway.java:1037, in org.flywaydb.core.Flyway.doValidate()

Exception: Java Exception

The above exception was the direct cause of the following exception:

org.flywaydb.core.api.FlywayException     Traceback (most recent call last)
File ~/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/ixmp/backend/jdbc.py:297, in JDBCBackend.__init__(self, jvmargs, **kwargs)
    296 try:
--> 297     self.jobj = java.Platform("Python", properties)
    298 except java.NoClassDefFoundError as e:  # pragma: no cover

org.flywaydb.core.api.FlywayException: org.flywaydb.core.api.FlywayException: Validate failed: Detected failed migration to version 1 (hsql base version)

During handling of the above exception, another exception occurred:

RuntimeError                              Traceback (most recent call last)
Cell In[2], line 2
      1 # launch the ix modeling platform using the default local database
----> 2 mp = ixmp.Platform()

File ~/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/ixmp/core/platform.py:94, in Platform.__init__(self, name, backend, **backend_args)
     88     raise ValueError(
     89         f"backend class {repr(backend_class_name)} not among "
     90         + str(sorted(BACKENDS.keys()))
     91     )
     93 # Instantiate the backend
---> 94 self._backend = backend_class(**kwargs)

File ~/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/ixmp/backend/jdbc.py:320, in JDBCBackend.__init__(self, jvmargs, **kwargs)
    318     else:
    319         _raise_jexception(e)
--> 320     raise RuntimeError(f"{msg}\n(Java: {jclass})")
    322 # Set the log level
    323 self.set_log_level(log_level)

RuntimeError: when initializing database:
(Java: org.flywaydb.core.api.FlywayException)
On: macos-latest-py3.8, macos-latest-py3.10

  • test_R_transport

    FAILED ixmp/tests/test_tutorials.py::test_R_transport - nbclient.exceptions.CellExecutionError: An error occurred while executing the following cell:
------------------
# launch the ix modeling platform using a local HSQL database instance
mp <- ixmp$Platform()
------------------


when initializing database:
(Java: org.flywaydb.core.api.FlywayException)Traceback:

1. ixmp$Platform()
2. py_call_impl(callable, call_args$unnamed, call_args$named)
On: macos-latest-py3.8, macos-latest-py3.10

  • test_R_transport_scenario

    FAILED ixmp/tests/test_tutorials.py::test_R_transport_scenario - nbclient.exceptions.CellExecutionError: An error occurred while executing the following cell:
------------------
# launch the ix modeling platform using the local default database
mp <- ixmp$Platform()
------------------


when initializing database:
(Java: org.flywaydb.core.api.FlywayException)Traceback:

1. ixmp$Platform()
2. py_call_impl(callable, call_args$unnamed, call_args$named)
On: macos-latest-py3.8, macos-latest-py3.10

  • test_del_ts

    FAILED ixmp/tests/backend/test_jdbc.py::test_del_ts - assert 8 == (1 + 8)
 +  where 8 = len(<WeakKeyDictionary at 0x120d52330>)
 +    where <WeakKeyDictionary at 0x120d52330> = <ixmp.backend.jdbc.JDBCBackend object at 0x14b0e3890>.jindex
 +      where <ixmp.backend.jdbc.JDBCBackend object at 0x14b0e3890> = <ixmp.core.platform.Platform object at 0x14b0e2bd0>._backend
On: macos-latest-py3.12

  • test_last_update[TimeSeries]

    FAILED ixmp/tests/core/test_timeseries.py::TestTimeSeries::test_last_update[TimeSeries] - assert datetime.timedelta(seconds=1, microseconds=390684) < datetime.timedelta(seconds=1)
 +  where datetime.timedelta(seconds=1, microseconds=390684) = abs((datetime.datetime(2024, 1, 12, 6, 36, 10, 712000) - datetime.datetime(2024, 1, 12, 6, 36, 12, 102684)))
 +    where datetime.datetime(2024, 1, 12, 6, 36, 12, 102684) = <built-in method now of type object at 0x109bc0ba8>()
 +      where <built-in method now of type object at 0x109bc0ba8> = datetime.now
 +  and   datetime.timedelta(seconds=1) = timedelta(seconds=1)
On: macos-latest-py3.12

  • test_export_timeseries_data

    FAILED ixmp/tests/core/test_platform.py::test_export_timeseries_data - AssertionError: DataFrame are different

DataFrame shape mismatch
[left]:  (2, 10)
[right]: (0, 10)
On: macos-latest-py3.8, macos-latest-py3.12

  • test_connect_message

    FAILED ixmp/tests/backend/test_jdbc.py::test_connect_message - assert "connected to database 'jdbc:hsqldb:mem:https://ixmptest' (user: ixmp)..." in ''
 +  where '' = CaptureResult(out='', err='').out
On: windows-latest-py3.9

  • test_del_ts

    FAILED ixmp/tests/backend/test_jdbc.py::test_del_ts - assert 8 == (1 + 8)
 +  where 8 = len(<WeakKeyDictionary at 0x1264df1a0>)
 +    where <WeakKeyDictionary at 0x1264df1a0> = <ixmp.backend.jdbc.JDBCBackend object at 0x151195730>.jindex
 +      where <ixmp.backend.jdbc.JDBCBackend object at 0x151195730> = <ixmp.core.platform.Platform object at 0x1511949e0>._backend
On: macos-latest-py3.12

  • test_close

    FAILED ixmp/tests/backend/test_jdbc.py::test_close - AssertionError: assert 'Database connection could not be closed or was already closed' in ''
 +  where '' = CaptureResult(out='', err='').out
On: windows-latest-py3.8

  • test_del_ts

    FAILED ixmp/tests/backend/test_jdbc.py::test_del_ts - AssertionError: ("<frame at 0x146cd3510, file '/Users/runner/work/ixmp/ixmp/ixmp/backend/jdbc.py', line 732, code check_out>
  <frame a..._>
  <frame at 0x1252f9e40, file '/Users/runner/work/ixmp/ixmp/ixmp/testing/data.py', line 220, code make_dantzig>", [])
assert 1 == (20 - 1)
 +  where 20 = getrefcount(<ixmp.core.scenario.Scenario object at 0x146b920f0>)
On: macos-latest-py3.12

  • test_platform_copy

    FAILED ixmp/tests/test_cli.py::test_platform_copy - AssertionError: Copy /tmp/pytest-of-runner/pytest-0/popen-gw0/test_platform_copy0/p2.script.new → /tmp/pytest-of-runner/pytest-0/popen-gw0/test_platform_copy0/p3.new
  Copy /tmp/pytest-of-runner/pytest-0/popen-gw0/test_platform_copy0/p2.properties.new → /tmp/pytest-of-runner/pytest-0/popen-gw0/test_platform_copy0/p3.new

assert (not True or 1 == 0)
 +  where 1 = <Result FileNotFoundError(2, 'No such file or directory')>.exit_code
On: ubuntu-latest-py3.12, macos-latest-py3.12

@glatterf42 glatterf42 linked a pull request Mar 12, 2024 that will close this issue
Open
2 tasks
@khaeru khaeru modified the milestones: 3.9, 3.10 Jun 4, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug ci Continuous integration
Projects
None yet
Milestone
3.10
Development

Successfully merging a pull request may close this issue.

Make tests robust & independent glatterf42/ixmp
2 participants
@khaeru @glatterf42