-
Notifications
You must be signed in to change notification settings - Fork 243
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
dumpfields=true fails in module_fcst_grid_comp.F90 for coupled model #1765
Comments
@DeniseWorthen Is this also failing with ESMF 8.5.0. @billsacks know that part of code better since he implemented multi-tile I/O support and could have some idea. |
This should work with recent versions of ESMF - or at least, recent versions of ESMF shouldn't give this particular error. Multi-tile I/O support was introduced in ESMF 8.4.0 with some limitations; some of these limitations were addressed in 8.5.0 and additional limitations were addressed in 8.6.0. |
@billsacks Thanks. That is really helpful. @DeniseWorthen It would be nice to test this again when new spack-stack (1.6.0, #2036) is available with ESMF 8.6.0. If we still issue, we could try to fix it. Anyway, let me know what you think? |
Thanks. My understanding was the way we were doing multi-tile output needed to be re-worked, so that the |
@DeniseWorthen May I ask if further code updates are required with ESMF 8.6.0? |
@junwang-noaa No, I think we need updates on the FV3 side that tries to use the multi-tiled IO. |
I ran the cpld_control_p8 test with dumpfields set to true, and I see the following error:
I added a single '*' to the file name and the code passed that point in module_fcst_grid_comp.F90 but is now crashing with the following error in module_cap_cpl.F90:
where is |
@DusanJovic-NOAA I can see it in the esmf code here src/Superstructure/IOAPI/interface/ESMFIO.F90 |
@DusanJovic-NOAA I don't have all the details but as I know those calls only used internally by ESMF. So, they are not exposed to user. They are internally called when you cal FieldWrite etc. (any call to write Field and Fieldbundle that is exposed to user), ESMF creates the ESMF I/O object to use PIO capability. Anyway, @billsacks could add more in here since he extended multi-tile I/O support in the ESMF side. |
Okay, thanks. Let me try to use FieldWrite instead. |
@DusanJovic-NOAA As I know Array write calls also use same underlying I/O infrastructure. So, If ArrayWrite etc. is failing there could be a bug in the ESMF side. There could be some limitations in writing the fields in ESMF side that I don't know. Again, @billsacks might have more information. Please open a support ticket if you think that this is bug. |
Interesting. This was a learning experience for me. It looks like ESMFIO.F90 is an entirely different, undocumented I/O interface that Raffaele Montuoro wrote in 2018 to enable I/O of multi-tile Fields. Unlike most ESMF I/O, this does not go through PIO, but instead calls into netcdf directly. I haven't read through it carefully, but my guess is that the functionality of this module may now be superseded by the multi-tile I/O work I did for FieldWrite, FieldBundleWrite, etc. |
I tried to convert the diagnose_cplFields routine (actually State_RWFields_tiles that is called by diagnose_cplFields) to use FieldBundleWrite instead of ESMFIO_Write, and now I see the following error:
|
It looks like you are running into a limitation with multi-tile I/O that was removed in the ESMF 8.6.0 release: prior to 8.6.0, multi-tile I/O only worked on Arrays / Fields with 1 DE per PET. Based on this error, it seems like you have Fields that use a decomposition with multiple DEs per PET (or possibly 0 DEs per PET). Solutions to this would be to either update to the 8.6.0 release (or the soon-upcoming 8.6.1 release that will have some other patches wanted by the UFS) or, if feasible, change the decomposition of these fields to always use the default of 1 DE per PET. |
Thanks @billsacks I'll try to build the model with ESMF 8.6.0 |
I updated diagnose_cplFields routine in FV3 to use ESMF_FieldBundleWrite. I can now write the coupling fields on 6-tiles with ESMF v8.6.0. Code is this branch: https://github.com/DusanJovic-NOAA/fv3atm/tree/dump_cpl_fields It can be tested using the corresponding ufs-weather-model branch: https://github.com/DusanJovic-NOAA/ufs-weather-model/tree/dump_cpl_fields I temporarily updated the esmf to version 8.6.0 from spack-stack 1.6.0 on Hera. |
When this makes it to PR form, make sure to add a dependency on ESMF 8.6.0 |
@DusanJovic-NOAA now we have ESMF 8.6.0 on all platforms, would you please test again to see if this issue is resolved? Thanks. |
Description
The nems.configure variable
dumpfields=true
should allow the coupling fields to be written from the component itself. This feature was previously working in the fv3_cap, but now fails with the following error:CMEPS has the capability in the mediator history files to write the fields it receives but it is sometimes useful to confirm that the fields imported by CMEPS are identical to those exported by FV3. This is no longer possible.
To Reproduce:
Run any of the global coupled configurations using the RT system. Use the run directory and set
DumpFields = true
in the ATM configuration attributes innems.configure
.Additional context
Output
The text was updated successfully, but these errors were encountered: