Optimize netcdf write component

[jswhit2]

Some modifications to the netcdf write component to improve IO speed and compression.

IO speed for uncompressed files has been improved by

1. making sure all attributes are written before nf90_enddef is called (rather than switching in and out of define model with nf90_enddef/nf90_redef).
2. creating the dataset with NF90_SHARE

C384 tests on hera show that these modifications improve the write speed by 4-5x (with compression turned off by setting ideflate=0 in model_configure). Write speeds with uncompressed netcdf are now very close to nemsio.

Unfortunately, this has no effect when compression is enabled. To improve write performance with compression I have tried tuning the chunksizes and chunk cache. The write speed seems very insensitive to these parameters, however the files compress better with a larger chunk size. Changing the chunk size to be the size of the full 3d state results in about a 25% reduction in the 3d file size with ideflate=1, nbits=14.

These changes are in PR #19

[jswhit2]

I may have overstated the compression improvements gained by increasing the chunksize - it appears that in many cases the improvements are not all that significant. However, it does speed up reading 2d horizontal slices of data quite significantly - which is a fairly common access pattern. Read speeds for 3d slices do not appear to be affected.

[jswhit2]

I have disabled writing of the _FillValue attribute, since the type of fill_value doesn't match the type of the variables in all cases this results in a 'fill value data type mismatch' error when you try to read the file. The default _FillValue is now used.

[junwang-noaa]

Jeff,

Are the code changes ready for commit for you are still working on it? Thanks.

[jswhit]

It's ready

[junwang-noaa]

Thanks, I put it in the fv3atm commit queue, it will be committed early Jan after CCPP code changes.

[junwang-noaa]

The code changes are committed, close issue.