[FEATURE REQUEST] Better documentation about boundary conditions and masks in for nested-grid simulations

[jennyfisher]

Request a new GEOS-Chem feature:

Hi there GCST,

We've just spent a lot of time debugging an emissions issue in our nested run. In the end it was (sort of) our fault, but I think some clarifications in the documentation (or possibly some automated checking if that is possible) would have prevented it.

The issue had to do with mask boundary definitions in HEMCO_Config.rc. It's actually described in a global context here: https://hemco.readthedocs.io/en/stable/hco-ref-guide/more-examples.html?highlight=defining%20masks#exercise-care-in-defining-mask-regions

From that page:

Long story short: if you set the mask range to a domain that is somewhat smaller than your simulation window, things work just fine. But if you set the range to something bigger, HEMCO will start ignoring emission files.

The issue we had was with a pre-defined mask in HEMCO_Config.rc, which was defined with boundaries that were larger than our nested region, triggering the issue described. It took a lot of digging to find this.

My suggestions would be:

• In the section linked above, add some text and/or an example of how this would play out in a nested simulation (at first we didn't think it applied because it was talking about global simulations).
• That section was pretty hard to find, so I would suggest also noting it briefly (with a link to the more detailed description/examples) on the page that talks about the HEMCO_Config.rc MASKS section.
• This is particularly likely to be a problem for nested grids now that we can use custom boundaries, because it's not obvious that we need to check the existing mask boundaries. So I would suggest some mention/links on any manual/wiki pages that explain how to set up a nested grid simulation. Even better if there is a message to check for this when you create a nested run directory.
• The best option would be if there is some error checking for this somewhere, probably when the run directory is created (but otherwise at runtime) that actually checks for nested runs whether any mask boundaries are larger than the simulation domain, and either prompts the user to fix them before continuing or issues a warning with info about what mask and what needs to be fixed.

Thanks!
-Jenny

[yantosca]

Thanks @jennyfisher for the feedback. Would you be able to provide examples that didn't work and that did work? We can incorporate that into the doc.

[jennyfisher]

Sure! Also tagging @smm997 as the one who's actually doing the work. :)

We're running a nested grid simulation over a non-standard Asian domain with longitude bounds 100.0-140.0 and latitude bounds 20.0 to 50.0.

In addition to global emissions, we have two Asian inventories that cover different regions. Example shown for one part of species TOLU from each inventory.

In the Base Emissions section:

0 CEDS_TOLU_AGR   $ROOT/CEDS/v2021-06/$YYYY/TOLU-em-anthro_CMIP_CEDS_$YYYY.nc       TOLU_agr          1750-2019/1-12/1/0 C xy kg/m2/s     TOLU  26               1 5
0 MEIC_TOLU_IND   $ROOT/MEIC_EI/MEIC_TOLU.05x0666.nc                                TOLU_industry     2000-2017/1-12/1/0 C xy kgC/m2/s    TOLU  302/307/1009     1/2 56
0 KORUS_TOLU_agts $ROOT/KORUS_EI/agts_l.2015$MM01.1.KORUS_01d.KORUSv5_01d.GC.nc     ARO1              2015/1-12/1/0      C xy molec/m2/s  TOLU  26/382/1006      1/2 55

In the Masks section:

1006 ASIA_MASK   $ROOT/MASKS/v2018-09/MIX_Asia_mask.generic.025x025.nc         MASK     2000/1/1/0 C xy 1 1 46/-12/180/82
1009 CHINA_MASK  $ROOT/MASKS/v2018-09/China_mask.generic.1x1.nc                MASK     2000/1/1/0 C xy 1 1 70/10/140/60

Because MEIC has a larger hierarchy (56) than KORUS (55), and both have a larger hierarchy than the CEDS global inventory (5), what should happen is that we get CEDS emissions globally, overwritten by KORUS in the 1006 mask region, further overwritten by MEIC in the (smaller) 1009 mask region. Indeed, this is what happens in our global simulations.

In our nested simulations, however, the HEMCO-defined mask region boundaries (i.e. 70/10/140/60 for the 1009 mask) extend beyond the bounds of our domain, so the simulation treats all 3 as effectively "global" and only reads/uses the emissions for the highest hierarchy (MEIC). That ends up being a problem as the actual China mask (the 1s in the file) is much smaller than the 70/10/140/60 rectangle and smaller than our domain, so we need the underlying emissions from the KORUS inventory for the rest of the domain (in our case we don't need CEDS - KORUS covers the region - so I'm ignoring that here, but it would have the same issue).

We suspected this was the root of the problem and found it was true when looking at a verbose HEMCO log that had a number of messages about skipping reading KORUS emissions because of MEIC.

The fix is to make the boundaries of the defined mask region at least a little bit smaller than the boundaries of the nested domain (which requires inspecting the mask itself to make sure we're not dropping gridboxes we shouldn't). In our case, we were able to fix by changing 70/10/140/60 to 70/10/136/60 in the 1009 mask line. Since our domain extends to 140E and the mask region now only extends to 136E, this was no longer seen as a global inventory.

I think there could easily be some automated error checking here, i.e. in this case identifying that these two masks were bigger than our domain and prompting the user to check this. I'm also thinking perhaps the defaults used in the HEMCO_Config.rc could be a bit more precise, going just to the edges of where the mask=1 rather than a semi-random choice about where to define the bounding box (most likely based on someone eyeballing a global map at some stage).

Hope that helps!

[yantosca]

Thanks @jennyfisher. In the future we can try to add better error checking but it may be a while before we can get to this.

[jennyfisher]

Thanks @yantosca - might be useful if the discussion today about adding KORUS + MEIC emissions inventories leads to both ending up in the model! :)

[yantosca]

Hi @jennyfisher @smm997, I've added a section to ReadTheDocs describing this issue (although I made it a more general w/o reference to KORUS or MEIC). See: https://hemco.readthedocs.io/en/latest/hco-ref-guide/hemco-config.html#caveat-for-simulations-using-cropped-horizontal-grids

This is in the "latest" version of ReadTheDocs (which is our docs/dev branch). When we release GEOS-Chem 14.1.0 and HEMCO 3.6.0 it will be in the "stable" documentation.

[yantosca]

I can close out this issue since this update is now in the GEOS-Chem 14.1.0 / HEMCO 3.6.0 development branch.

[jennyfisher]

Thanks @yantosca !

[yantosca]

@jennyfisher: Further developments will be tracked at geoschem/HEMCO#187