New write mode to append to existing netCDF file

[sadielbartholomew]

Fixes #30 by enabling the new append mode feature for writing to netCDF as implemented in NCAS-CMS/cfdm#69 to be used here in cf-python.

[sadielbartholomew]

Actions jobs will currently fail as I am investigating a few issues that have arisen on this side of play...

[sadielbartholomew]

(Force-pushed the branch to resolve minor conflicts.)

[sadielbartholomew]

I am seeing a (very unhelpful) generic HDF error when the test hits the "NETCDF4" format, which I haven't seen before in the equivalent cdfm test:

...
>>>>>>>>>>> ONTO FIELD N = 0 for NETCDF4
ERROR
test_write_reference_datetime (__main__.read_writeTest) ... ok

======================================================================
ERROR: test_write_netcdf_mode (__main__.read_writeTest)
Test the `mode` parameter to `write`, notably append mode.
----------------------------------------------------------------------
Traceback (most recent call last):
  File "test_read_write.py", line 297, in test_write_netcdf_mode
    cf.write(ex_field, tmpfile, fmt=fmt, mode="a")
  File "/home/sadie/cfdm/cfdm/decorators.py", line 181, in verbose_override_wrapper
    return method_with_verbose_kwarg(*args, **kwargs)
  File "/home/sadie/cf-python/cf/read_write/write.py", line 703, in write
    netcdf.write(
  File "/home/sadie/cfdm/cfdm/decorators.py", line 181, in verbose_override_wrapper
    return method_with_verbose_kwarg(*args, **kwargs)
  File "/home/sadie/cfdm/cfdm/read_write/netcdf/netcdfwrite.py", line 4733, in write
    return self._file_io_iteration(
  File "/home/sadie/cfdm/cfdm/read_write/netcdf/netcdfwrite.py", line 4935, in _file_io_iteration
    g["netcdf"] = self.file_open(filename, mode, fmt, fields)
  File "/home/sadie/cfdm/cfdm/read_write/netcdf/netcdfwrite.py", line 4376, in file_open
    nc = netCDF4.Dataset(filename, mode, format=fmt)
  File "src/netCDF4/_netCDF4.pyx", line 2330, in netCDF4._netCDF4.Dataset.__init__
  File "src/netCDF4/_netCDF4.pyx", line 1948, in netCDF4._netCDF4._ensure_nc_success
OSError: [Errno -101] NetCDF: HDF error: b'/home/sadie/cf-python/cf/test/tmp5w7f4l2u_test_read_write.nc'

----------------------------------------------------------------------
Ran 17 tests in 13.934s

FAILED (errors=1)

Investigating...

[sadielbartholomew]

Opening and closing to trigger the Actions jobs to see see whether the test suite passes at this point...

[sadielbartholomew]

The Actions jobs are failing currently due to using the current cfdm master where the API change to allow for a mode parameter to write is not included. That cfdm-side PR is about to be merged, then once a new commit or two is added to fix the one remaining issue, to be detailed and fixed here shortly, we can run the jobs for a proper test of the PR.

[sadielbartholomew]

RE #213 (comment), on Friday I managed to extract a more specific HDF error message, as follows:

HDF5-DIAG: Error detected in HDF5 (1.10.4) thread 139673628755776:
  #000: H5F.c line 509 in H5Fopen(): unable to open file
    major: File accessibilty
    minor: Unable to open file
  #001: H5Fint.c line 1400 in H5F__open(): unable to open file
    major: File accessibilty
    minor: Unable to open file
  #002: H5Fint.c line 1583 in H5F_open(): file is already open for read-only
    major: File accessibilty
    minor: Unable to open file

which pointed to opening files that were already open in some mode as the issue at hand. After discussions today we found a probably solution and realised the issue applied only to cf-python (i.e. not cfdm) because of its different file handler management approach hence why we were seeing this only via the tests here and not in the tests on NCAS-CMS/cfdm#69.

Fix to be committed shortly...

[sadielbartholomew]

Looks there are a few linting issues to resolve, and I should bump the release date, I'll do that in the next commit, but it would be good to let a test suite job run to completion to see if there are any issues raised. I may have seen one (outside of the test_read_write module) locally.

[sadielbartholomew]

I may have seen one (outside of the test_read_write module) locally.

In fact that was just due to pycodestyle not being installed after having blitzed my conda environment trying to get to the bottom of packaging issues. Have a full local pass, though linting is only done on a per-commit basis for files touched so that failure wasn't detected.

[sadielbartholomew]

I have added a few pre-release commits though they are not related to the append mode crux of this PR, because I want to use the Actions jobs as checks of the codebase for the imminent release. Restarting the jobs with an open-close...

[sadielbartholomew]

The test_source_dist job is failing but it is not clear why from the logs, at least as far as I can tell. I will investigate that as part of the pre-release checklist. Also missed a few linting issues because I only checked the cf/ directory. Going again...

[sadielbartholomew]

Some of the test suite run jobs appear to have hung and there is not much I can do about that (bar re-start the jobs but we might hit the same issue and waste much limited Actions time resource), so going to merge. The test suite passes locally and has passed for a Mac OS and an Ubuntu job, so I think that is good enough.