Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Lazy netcdf saves #5191

Merged
merged 92 commits into from
Apr 21, 2023
Merged

Lazy netcdf saves #5191

Changes from 1 commit
Commits
Show all changes
92 commits
Select commit Hold shift + click to select a range
cd7fa42
Basic functional lazy saving.
pp-mo Oct 21, 2022
1f32800
Simplify function signature which upsets Sphinx.
pp-mo Oct 21, 2022
e0f980f
Non-lazy saves return nothing.
pp-mo Oct 21, 2022
67b96cf
Now fixed to enable use with process/distributed scheduling.
pp-mo Oct 23, 2022
8cdbc9b
Remove dask.utils.SerializableLock, which I think was a mistake.
pp-mo Mar 3, 2023
8723f24
Make DefferedSaveWrapper use _thread_safe_nc.
pp-mo Mar 8, 2023
d19a87f
Fixes for non-lazy save.
pp-mo Mar 9, 2023
45e0e60
Avoid saver error when no deferred writes.
pp-mo Mar 10, 2023
6a83200
Reorganise locking code, ready for shareable locks.
pp-mo Mar 10, 2023
78a9346
Remove optional usage of 'filelock' for lazy saves.
pp-mo Mar 10, 2023
31b12f7
Document dask-specific locking; implement differently for threads or …
pp-mo Mar 10, 2023
99b4f41
Minor fix for unit-tests.
pp-mo Mar 10, 2023
5d0a707
Pin libnetcdf to avoid problems -- see #5187.
pp-mo Mar 10, 2023
431036f
Minor test fix.
pp-mo Mar 10, 2023
dc368d9
Move DeferredSaveWrapper into _thread_safe_nc; replicate the NetCDFDa…
pp-mo Mar 13, 2023
6756a46
Update lib/iris/fileformats/netcdf/saver.py
pp-mo Mar 16, 2023
80b4b6c
Update lib/iris/fileformats/netcdf/_dask_locks.py
pp-mo Mar 16, 2023
78a8716
Update lib/iris/fileformats/netcdf/saver.py
pp-mo Mar 16, 2023
47bb08b
Small rename + reformat.
pp-mo Mar 17, 2023
0ece09a
Remove Saver lazy option; all lazy saves are delayed; factor out fill…
pp-mo Mar 18, 2023
940f544
Merge branch 'main' into lazy_save_2
pp-mo Mar 18, 2023
eb97130
Merge remote-tracking branch 'upstream/main' into lazy_save_2
pp-mo Mar 20, 2023
4596081
Repurposed 'test__FillValueMaskCheckAndStoreTarget' to 'test__data_fi…
pp-mo Mar 20, 2023
ad49fbe
Disable (temporary) saver debug printouts.
pp-mo Mar 20, 2023
b29c927
Fix test problems; Saver automatically completes to preserve existing…
pp-mo Mar 20, 2023
8f10281
Fix docstring error.
pp-mo Mar 20, 2023
6a564d9
Fix spurious error in old saver test.
pp-mo Mar 20, 2023
2fb4d6c
Fix Saver docstring.
pp-mo Mar 20, 2023
c84bfdc
More robust exit for NetCDFWriteProxy operation.
pp-mo Mar 20, 2023
5b78085
Fix doctests by making the Saver example functional.
pp-mo Mar 21, 2023
478332e
Improve docstrings; unify terminology; simplify non-lazy save call.
pp-mo Mar 23, 2023
34f154c
Moved netcdf cell-method handling into nc_load_rules.helpers, and var…
pp-mo Mar 27, 2023
d3744ba
Merge branch 'latest' into lazy_save_2
pp-mo Mar 27, 2023
9673ea0
Fix lockfiles and Makefile process.
pp-mo Mar 27, 2023
bcbcbc8
Add unit tests for routine _fillvalue_report().
pp-mo Mar 27, 2023
05c04a1
Remove debug-only code.
pp-mo Mar 27, 2023
679ea47
Added tests for what the save function does with the 'compute' keyword.
pp-mo Mar 28, 2023
70ec9dd
Fix mock-specific problems, small tidy.
pp-mo Mar 28, 2023
28a4674
Restructure hierarchy of tests.unit.fileformats.netcdf
pp-mo Mar 29, 2023
67f4b2b
Tidy test docstrings.
pp-mo Mar 29, 2023
ebec72f
Correct test import.
pp-mo Mar 29, 2023
1f5b904
Avoid incorrect checking of byte data, and a numpy deprecation warning.
pp-mo Mar 29, 2023
5045c9f
Alter parameter names to make test reports clearer.
pp-mo Mar 29, 2023
393407a
Test basic behaviour of _lazy_stream_data; make 'Saver._delayed_write…
pp-mo Mar 29, 2023
518360b
Add integration tests, and distributed dependency.
pp-mo Mar 30, 2023
5c9931f
Docstring fixes.
pp-mo Mar 31, 2023
7daee68
Documentation section and whatsnew entry.
pp-mo Apr 4, 2023
97474f9
Merge branch 'main' into lazy_save_2
pp-mo Apr 4, 2023
64c7251
Various fixes to whatsnew, docstrings and docs.
pp-mo Apr 4, 2023
75043f9
Minor review changes, fix doctest.
pp-mo Apr 11, 2023
445fbe2
Arrange tests + results to organise by package-name alone.
pp-mo Apr 11, 2023
09cb22e
Review changes.
pp-mo Apr 11, 2023
3445f58
Review changes.
pp-mo Apr 12, 2023
cb1e1f7
Enhance tests + debug.
pp-mo Apr 12, 2023
1c81cee
Support scheduler type 'single-threaded'; allow retries on delayed-sa…
pp-mo Apr 13, 2023
370837b
Improve test.
pp-mo Apr 13, 2023
2f5f3c2
Adding a whatsnew entry for 5224 (#5234)
HGWright Apr 4, 2023
a55c6f2
Replacing numpy legacy printing with array2string and remaking result…
HGWright Apr 4, 2023
4914e99
adding a whatsnew entry
HGWright Apr 4, 2023
bd642cd
configure codecov
HGWright Apr 4, 2023
bc5bdd1
remove results creation commit from blame
HGWright Apr 4, 2023
301e59e
fixing whatsnew entry
HGWright Apr 4, 2023
7b3044d
Bump scitools/workflows from 2023.04.1 to 2023.04.2 (#5236)
dependabot[bot] Apr 5, 2023
02f2b66
Use real array for data of of small netCDF variables. (#5229)
pp-mo Apr 6, 2023
a7e0689
Handle derived coordinates correctly in `concatenate` (#5096)
schlunma Apr 12, 2023
c4e8bbb
clarity on whatsnew entry contributors (#5240)
bjlittle Apr 12, 2023
e6661b8
Modernize and simplify iris.analysis._Groupby (#5015)
bouweandela Apr 12, 2023
afbdbbd
Finalises Lazy Data documentation (#5137)
ESadek-MO Apr 12, 2023
b8bb753
Fixes to _discontiguity_in_bounds (attempt 2) (#4975)
stephenworsley Apr 12, 2023
97cc149
update ci locks location (#5228)
bjlittle Apr 13, 2023
f14a321
Updated environment lockfiles (#5211)
scitools-ci[bot] Apr 13, 2023
f7a0b87
Increase retries.
pp-mo Apr 13, 2023
69ddd9d
Change debug to show which elements failed.
pp-mo Apr 13, 2023
8235d60
update cf standard units (#5244)
ESadek-MO Apr 13, 2023
724c6d2
libnetcdf <4.9 pin (#5242)
trexfeathers Apr 13, 2023
4f50dc7
Avoid possible same-file crossover between tests.
pp-mo Apr 13, 2023
0da68cf
Ensure all-different testfiles; load all vars lazy.
pp-mo Apr 13, 2023
e8b7bfd
Revert changes to testing framework.
pp-mo Apr 13, 2023
ad48caf
Remove repeated line from requirements/py*.yml (?merge error), and re…
pp-mo Apr 13, 2023
291b587
Revert some more debug changes.
pp-mo Apr 13, 2023
b2260ef
Merge branch 'latest' into lazy_save_2
pp-mo Apr 13, 2023
33a7d86
Reorganise test for better code clarity.
pp-mo Apr 14, 2023
db6932d
Use public 'Dataset.isopen()' instead of '._isopen'.
pp-mo Apr 14, 2023
631e001
Create output files in unique temporary directories.
pp-mo Apr 14, 2023
2869f97
Tests for fileformats.netcdf._dask_locks.
pp-mo Apr 14, 2023
419727b
Merge branch 'latest' into lazy_save_2
pp-mo Apr 21, 2023
2f4458b
Fix attribution names.
pp-mo Apr 21, 2023
88b7a2a
Merge branch 'latest' into lazy_save_2
pp-mo Apr 21, 2023
98a20e7
Fixed new py311 lockfile.
pp-mo Apr 21, 2023
bbc1167
Fix typos spotted by codespell.
pp-mo Apr 21, 2023
ed38e43
Add distributed test dep for python 3.11
pp-mo Apr 21, 2023
54ec0f8
Fix lockfile for python 3.11
pp-mo Apr 21, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Fix attribution names.
  • Loading branch information
@pp-mo
pp-mo committed Apr 21, 2023
commit 2f4458bdbad24fb30b536161312979332e3f665d
3 changes: 2 additions & 1 deletion docs/src/whatsnew/latest.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -115,7 +115,8 @@ This document explains the changes made to Iris for this release
Whatsnew author names (@github name) in alphabetical order. Note that,
core dev names are automatically included by the common_links.inc:


.. _@fnattino: https://github.com/fnattino
.. _@tinyendian: https://github.com/tinyendian


.. comment
Expand Down