The re-design of config_src/

[adcroft]

The current directory tree of compilable source code is

.
|-- config_src
|   |-- coupled_driver
|   |-- dynamic
|   |-- dynamic_symmetric
|   |-- external
|   |   |-- GFDL_ocean_BGC
|   |   `-- ODA_hooks
|   |-- ice_solo_driver
|   |-- mct_driver
|   |-- nuopc_driver
|   |-- solo_driver
|   `-- unit_drivers
|-- docs
|-- pkg
|   |-- CVMix-src
|   `-- GSW-Fortran
`-- src

where the some of the cirectories within config_src/ are exclusive. e.g. you should only include either dynamic/ or dynamic_symmetric/ in the search path for the compiler. Similarly only one of the *_driver/ directories should be used, and the different choices yield different forms of the model.

We wish to add directories containing a thin layer that translates between a MOM6-defined API and infrastructure libraries. We envision different directories corresponding to different versions of infrastructure, initially fms1 and fms2, and later someone can add one for esmf. On the dev call we just discussed that rather than add yet more directories to config_src/ we should re-structure into something like:

.
|-- config_src
|   |--drivers
|   |   |-- coupled_driver
|   |   |-- ice_solo_driver
|   |   |-- mct_driver
|   |   |-- nuopc_driver
|   |   |-- solo_driver
|   |   `-- unit_drivers
|   |-- external
|   |   |-- GFDL_ocean_BGC
|   |   `-- ODA_hooks
|   |   |-- infrastructure
|   |   |   |-- FMS1
|   |   |   |-- FMS2
|   |   |   |-- ESMF
|   |   |   `-- nompi
|   |-- memory
|   |   |-- dynamic_nonsymmetric
|   |   `-- dynamic_symmetric
|-- docs
|-- pkg
|   |-- CVMix-src
|   `-- GSW-Fortran
`-- src

Better yet would be to group the infrastructure by operation. Proposals welcome...

[Hallberg-NOAA]

I would prefer not to have infrastructure under external, but I would promote it up one level to be at the same level as drivers and memory, to avoid excessively long pathnames and to reflect that at least one infrastructure directory is needed, whereas it is not necessary to use any of the other directories under external.

[adcroft]

Ah! That wasn't my intent. Making trees by hand is harder than making the actual tree of directories! I meant like this:

.
|-- config_src
|   |-- drivers
|   |   |-- coupled_driver
|   |   |-- ice_solo_driver
|   |   |-- mct_driver
|   |   |-- nuopc_driver
|   |   |-- solo_driver
|   |   `-- unit_drivers
|   |-- external
|   |   |-- GFDL_ocean_BGC
|   |   `-- ODA_hooks
|   |-- infrastructure
|   |   |-- FMS1
|   |   |-- FMS2
|   |   |-- ESMF
|   |   `-- nompi
|   |-- memory
|   |   |-- dynamic_nonsymmetric
|   |   `-- dynamic_symmetric
|-- docs
|-- pkg
|   |-- CVMix-src
|   `-- GSW-Fortran
`-- src

[klindsay28]

Is the driver suffix on the subdirectories of drivers needed?
It feels redundant, except perhaps for unit_drivers (which could be renamed unit_tests).

[alperaltuntas]

This makes perfect sense. No objections. One minor suggestion I have is to drop the "_driver" suffixes, since they are now redundant, I guess

.
|-- config_src
|   |-- drivers
|   |   |-- coupled
|   |   |-- ice_solo
|   |   |-- mct
|   |   |-- nuopc
|   |   |-- solo
|   |   `-- unit

[alperaltuntas]

@klindsay28 beat me to it.

[Hallberg-NOAA]

Dropping the _driver suffix is a good suggestion.

[sanAkel]

👍

[marshallward]

Once we get something working, a potential phase 2:

.
|-- config_src
|   |-- drivers
|   |   |-- coupled
|   |   |-- ice_solo
|   |   |-- mct
|   |   |-- nuopc
|   |   |-- solo
|   |   `-- unit
|   |-- external
|   |   |-- GFDL_ocean_BGC
|   |   `-- ODA_hooks
|   |-- calendar   # Calendar, model time computation, astronomy
|   |    |-- fms
|   |    `-- esmf
|   |-- io  # Logging, File I/O, etc
|   |   |-- binary
|   |   |-- netcdf
|   |   |-- fms1
|   |   |-- fms2
|   |   `-- pio
|   |-- mpp
|   |   |-- serial   # No parallelization
|   |   |-- mpi      # Native MPI calls
|   |   |-- fms      # FMS parallelization (whatever it may be)
|   |-- tracer       # Generic tracer ops
|   |    `-- fms
|   |-- diagnostic   # diagnostic manager
|   |    `-- fms
|   |-- ???
|   |-- memory
|   |   |-- dynamic_nonsymmetric
|   |   `-- dynamic_symmetric
|-- docs
|-- pkg
|   |-- CVMix-src
|   `-- GSW-Fortran
`-- src

[marshallward]

Also, I feel obliged to mention that submodules were introduced to address this problem.

They are not necessary to achieve what we want, but they would allow us to compile the src content and produce binaries and .mod files independently of config_src. It could also allow us to "hot swap" more easily between implementations. Without submodules, there will be a strong compile-time link between src and config_src content. Submodules replace this with a one-way dependency (config_src on src only).

One downside is that many compilers do not have a long history of support for them, many having only recently added them.

I don't think we need to pursue this, but wanted to at least make sure it was part of the discussion.

[adcroft]

1. Should io, mpp, tracer, diagnostic be grouped under infrastructure/
2. ??? should include timers (cpu/perf clocks). Not sure where time_interp or horiz_interp belong?
3. Do we need a misc for everything else like MersenneTwistor, constants, ... ?
4. What is tracer ? Haven't seen it before...
5. Should terminal i/o (logging) be separate from binary files (netcdf)?

[marshallward]

Sorry, that wasn't meant to be an exhaustive list, just an indication that infrastructure could be replaced with a set of particular components.

(I just grepped for non-MOM modules and looked at the various subroutines and variables, including generic_tracer_mod).

[sanAkel]

All of this opens up lots of possibilities. Once all sorted out and ready to go!

1. Whenever there is a PR, will it have to be tested for ALL infrastructures?
2. Let's say someone develops another option, say for diagnostics, whose burden would it be maintain it? The one who developed it or volunteers?

[marshallward]

1. When we discussed this internally, we proposed that GFDL would confirm that FMS2 works across all tests, but would commit to supporting and fixing FMS1 when issues become known.

   I think we'll still make our best effort to support as many frameworks as possible, however. I would also expect other labs to test their own infrastructure(s) of choice.

2. I personally I think the contributor should be responsible for the content, and If the contributor isn't willing to either review the code or keep it up to date, then we may want to reconsider accepting it.

   At the least, we should have a designated maintainer in mind for all future contributions.

[sanAkel]

Sounds good!

On 2, I would add that the designated maintainer also keep compiler flags also up to date, at least for some recent version of intel and or GNU?

[Hallberg-NOAA]

I have some questions that I would like to raise about documenting the interfaces with this new structure for FMS2:

• Should we document the interfaces that will come from the config_src/infra/.../ code in the code that will stay in the MOM6/src/framework code, or can we just rely on having them documented in the (eventually numerous?) versions that will appear in the config_src/infra/.../ code?

• If the latter option (documentation in config_src/infra/...), how do we want to handle any divergences between the documentation of the various versions? (The compilers and appropriate testing should keep the interfaces themselves consistent.)

• Do we need to add explicit interfaces to the config_src/infra/.../*.F90 code to document every interface inherited from FMS and eventually the other infrastructures?

[Hallberg-NOAA]

Upon reflection, we think that (1) every API offered to MOM6 code needs an explicit implementation in MOM6 code (perhaps just a very thin wrapper) - there will be no pass-throughs from the infrastructure to anything used by the MOM6/src code, and (2) each of these implementations is the right place for the definitive documentation. We will use diffs to keep the APIs in the various config_src/infra directories consistently documented.

[adcroft]

We've realized it is very hard to separate the io and communications from each other. When you combine IO with comms, and see what's left, almost everything is interconnected and the point of the fine grain directory structure is lost. The only things that look self contained are things like calendar, but even that is then used by time_interp. We are thus thinking we might stay with something more like in config_src/infrastructure/FMS1 form, as in https://github.com/NOAA-GFDL/MOM6/discussions/1286#discussioncomment-275600, and not the "phase 2" with more granularity. Make sense?

[Hallberg-NOAA]

After reflecting on various proposals and considering how they might actually be implemented I think that we are approaching a consensus for the new structure. The new proposal is for the new structure to be the following:

|-- config_src
|   |--drivers
|   |   |-- FMS_cap
|   |   |-- ice_solo_driver
|   |   |-- mct_cap
|   |   |-- nuopc_cap
|   |   |-- (python_cap?)
|   |   |-- solo_driver
|   |   `-- unit_drivers
|   |-- external
|   |   |-- GFDL_ocean_BGC
|   |   `-- ODA_hooks
|   |-- infra
|   |   |-- FMS1
|   |   |-- FMS2
|   |   |-- (ESMF?)
|   |   `-- (nompi?)
|   `-- memory
|       |-- dynamic_nonsymmetric
|       `-- dynamic_symmetric
|-- docs
|-- pkg
|   |-- CVMix-src
|   `-- GSW-Fortran
`-- src
    |-- ALE
    |-- core
    |-- diagnostics
    |-- eqn_of_state
    |-- framework
    |-- initialization
    |-- ocean_data_assim
    |-- parameterizations
    |-- tracer
    `-- user

The directories in parentheses do not exist, but are just placeholders for things that someone might want to add later.
Under drivers, a cap is something that acts to connect MOM6 with a coupler or external driving software (e.g., python), whereas a driver is code with its own Fortran main(). The contents of the current config_src/coupled_driver would be moved to config_src/drivers/FMS_cap, reflecting the fact that this is driven by the GFDL / FMS coupler code.

We are likely to start implementing this for a PR by mid-February, so any further comments by then would be appreciated.

[alperaltuntas]

Great, thanks!

[sanAkel]

@alperaltuntas do you happen to have pseudo code for what you have in mind? may be a gist if it is long?

[alperaltuntas]

@sanAkel, We don't have a concrete plan yet, but, as a first step, I am imagining a lightweight FMS patch to implement few CESM-specific functionalities like PIO interface and a CESM/ESMF-integrated global constants, logging/stdout implementations, etc. @Hallberg-NOAA's second characterization seems a more likely path (i.e., interface changes and so a new directory under infra).

[sanAkel]

Thank you @alperaltuntas. If possible, may be we could chat offline. My thoughts are bit sketchy right now, but we could follow your approach for

CESM/ESMF-integrated global constants, logging/stdout implementations,

of course with GEOS-MAPL constants, for instance, see https://github.com/GEOS-ESM/MAPL/blob/main/shared/MAPL_Constants.F90 as a variant from our side.

I can only imagine that the infra interface changes would be more stronger with a diverse pool! 🌞

[alperaltuntas]

@sanAkel Sure! let's talk and identify any common goals/requirements and address them in coordination wherever possible.