Run web_api tests with PYTHONASYNCIODEBUG=1

[sk1p]

Fix snooze task cancellation being called from the wrong thread (which was caught by this change).

Contributor Checklist:

• I have added or updated my entry in the creators.json file
• I have added a changelog entry for my contribution
• I have added/updated documentation for all user-facing changes
• I have added/updated test cases
• I have included the rebuilt production build of the client (only if changes were made to the GUI)

Reviewer Checklist:

• /azp run libertem.libertem-data passed
• No import of GPL code from MIT code

[sk1p]

/azp run libertem.libertem-data

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Comparison is base (01feaae) 70.50% compared to head (a8ffbda) 70.25%.
Report is 5 commits behind head on master.

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #1580      +/-   ##
==========================================
- Coverage   70.50%   70.25%   -0.25%     
==========================================
  Files         324      324              
  Lines       26873    26874       +1     
  Branches     3072     3072              
==========================================
- Hits        18946    18881      -65     
- Misses       7374     7430      +56     
- Partials      553      563      +10     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[matbryan52]

Reproduced what is probably the nbval test failure outside of tox:

Seems to be slow coroutines being logged as warning by the new envvar, which is bizarre as the variable is only set during the web_api env ??

NOTE: there seem to be two CI failures, first the async notebook fails from cell [2], then the job got stuck and later cancelled when running binning.ipynb.

[sk1p]

Reproduced what is probably the nbval test failure outside of tox:

[...]

Seems to be slow coroutines being logged as warning by the new envvar, which is bizarre as the variable is only set during the web_api env ??

Thanks for tracking this down. That is weird indeed - can you check os.environ['PYTHONASYNCIODEBUG'] inside the notebook? It's either this environment variable, loop.set_debug(True) or the Python development mode that should trigger these log messages...

The log messages themselves are related to long running tasks, see also #1577 - are there any other messages in there, maybe related to thread safety or other "async sins"?

NOTE: there seem to be two CI failures, first the async notebook fails from cell [2], then the job got stuck and later cancelled when running binning.ipynb.

[matbryan52]

Think it might just have been a red herring ? I started going down the rabbit hole of envvars bleeding through tox configs but actually it passed fine on re-run.

Maybe the worker had a hiccup which caused a log message in the notebook cell ?

I would say LGTM and monitor during other CI runs...

[sk1p]

Think it might just have been a red herring ? I started going down the rabbit hole of envvars bleeding through tox configs but actually it passed fine on re-run.

Maybe the worker had a hiccup which caused a log message in the notebook cell ?

I would say LGTM and monitor during other CI runs...

Thanks for looking into that. Yeah, I would concur. I'm quite certain that there are issues with async, especially if distributed is not up to date, or in the shutdown code, but this change should only affect the web API, not the bare Python API etc.

I'll go ahead and merge!

[sk1p]

I think I understand what was happening - sadly, there's some instability with the azure pipeline agents on our own infra, which means they "run out of space" (I think on /dev/shm or something like that?) after running a bunch of jobs. I didn't have time to properly diagnose that specific issue, but it may have to do with zombie Python processes that hang around longer than they should, and the pipeline agent not implementing proper isolation between runs. Right now my "solution" is to restart the containers...

If that happens during a test run, we might get these strange failures or hangs. I apologize that this sent you down a rabbit hole...