Fix regrid memory leak

[ThibHlln]

resolve #222

The special methods __del__ of ESMF.Field and ESMF.Grid objects feature calls to their respective destroy methods that release the memory allocated by ESMF in Fortran. When these objects go out of scope, the Python garbage collection ought to call these __del__ methods, yet their allocated memory does not seem to be released properly, as repeatedly regridding fields results in running out of memory after a while.

After a discussion with @davidhassell, we suspect that the ESMF.Manager may be responsible for not making the memory release possible as long as it stays in scope. However, in cf the Manager is initialised but not bound to a variable name (see e.g. https://github.com/ThibHlln/cf-python/blob/d9cde28d9abd9cfeb6575c737560956f40620942/cf/field.py#L15619), so to me it goes out of scope directly in a Python sense. The ESMF documentation says that "The Manager will be created when the first ESMPy object is created if it is not created explicitly by the user". So I suspect that the Manager stays around (as a Fortran object I suppose) even if not bound to a name in Python's namespace (which may also very well be the case for Fields/Grids objects then).

In the cf regridding, there are ESMF fields and grids created during each call to regrids/regridc. If these are explicitly releasing memory (through calls to their destroy methods), this does seem to eliminate the running out of memory when repeatedly calling the regridding methods. Note, calling the destroy methods on these objects several times does not raise an error, if memory was already released, it remains silent. So this PR does just that, i.e. it explicitly releases memory for all ESMF objects as soon as they are not required anymore. This does not provide as explanation for why the destroy calls in Python's garbage collection are not working/not enough, but it seems to be a cheap and harmless solution that solves the problem.

I am attaching these field files if you'd like to test the following script:

fld_src = cf.read('fld_src_3d.nc')[0]  # try also with 'fld_src_2d.nc'
fld_dst = cf.read('fld_dst_3d.nc')[0]  # try also with 'fld_dst_2d.nc'

regrid_op = fld_src.regrids(fld_dst, 'conservative', return_operator=True)
print('regrid weights generated')

for _ in range(5000):
    fld_src.regrids(regrid_op)

Before this PR, after the expensive generation of the weights (it may take a short while), the memory was building up very quickly in the loop until running out of memory (using 8GB of RAM for my test), while with the modifications in this PR the memory now remains stable across the loop.

[davidhassell]

Thanks, @ThibHlln - I look forward to looking at this next week. Have a good weekend,
David

[davidhassell]

Hi Thibault,

This all looks sound, along the lines we have discussed. I confirm the behaviour you report with this PR vs. 3.10.0.

I've noted couple of small things we should look at before we merge.

Thanks for sorting this out - I especially appreciate you taking the time ensure that everything is consistent by modifying files in cf/regrid/.

Some of these changes will need to be copied in cf.Field.regridc, and you are welcome to add those in :), but equally I'm happy to do that in a separate PR after this has been merged.

David

[ThibHlln]

Hi David,

Thank you for your careful review.
I believe everything done in regrids is done in regridc too now, although I must say I have not run any test using regridc yet.
I could run those tests now, if you have not done it yet.

T

[davidhassell]

Thanks - looks good.

[sadielbartholomew]

@davidhassell - when you return from leave - is there a reason this hasn't been merged despite looking like it is ready / in a mergable state? We don't want to keep Thibault waiting if there isn't a reason. Thanks.

[davidhassell]

Ah - no reason! I'll merge it now. Thanks.