[BUG/ISSUE] Inconsistent file target for error messages

[lizziel]

We recently had a user help request that had HEMCO error messages in both HEMCO.log and the GEOS-Chem log. The HEMCO.log file had this:

!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
HEMCO ERROR: Invalid time index in /home/disk/oreo/data/ctm/HEMCO/CMIP6/v2019-12/CMIP6_GHG_surface_VMR_2014.nc
ERROR LOCATION: HCOIO_READ_STD (hcoio_read_std_mod.F90)
ERROR LOCATION: HCOIO_DataRead (hcoio_dataread_mod.F90)
ERROR LOCATION: ReadList_Fill (hco_readlist_mod.F90)
ERROR LOCATION: ReadList_Read (hco_readlist_mod.F90)
ERROR LOCATION: HCO_RUN (hco_driver_mod.F90)

The GEOS-Chem log had this:

 Error in HCOIO_DATAREAD called from HEMCO ReadList_Fill: SfcVMR_CH3Cl
 Error in ReadList_Fill (3) called from HEMCO ReadList_Read
 Error in ReadList_Read called from hco_run
=====================================================================
GEOS-Chem ERROR: Error encountered in "HCO_Run"!
 -> at HCOI_GC_Run (in module GeosCore/hcoi_gc_main_mod.F90)
THIS ERROR ORIGINATED IN HEMCO!  Please check the HEMCO log file for 
additional error messages!
=====================================================================
=====================================================================
GEOS-Chem ERROR: Error encountered in "HCOI_GC_Run"!
 -> at Emissions_Run (in module GeosCore/emissions_mod.F90)
=====================================================================
=====================================================================
GEOS-CHEM ERROR: Error encountered in "Emissions_Run", Phase 0
This error can indicate a missing file. Please check the HEMCO log file for 
additional error messages!
STOP at  -> at GEOS-Chem (in GeosCore/main.F)
=====================================================================

I think this problem was introduced when I recently added more error messages to HEMCO. That code needs to be updated to send errors to the HEMCO log rather than GEOS-Chem log.

[jimmielin]

Hi Lizzie,

It seems that this is caused because it's printing to standard output:

       IF ( RC /= HCO_SUCCESS ) THEN
          PRINT *, "Error in ReadList_Fill (1) called from HEMCO ReadList_Read"
          RETURN
       ENDIF

It should be an easy fix for it to call HCO_MSG too. Note that the print or write to stdout are potentially dangerous in a distributed environment, as these calls don't check am_I_Root from what I've seen.

I've searched the print statements in HEMCO and attached a list, hope this is helpful. I can also help cleaning some of these as I go through the files.
print_hemco.txt

[lizziel]

Thanks @jimmielin. Yes, the fix should be straightforward. I made this issue mainly to remind myself to go back and apply it. Feel free to tackle this!

[yantosca]

This is related to the following issues:

• [BUG/ISSUE] Full-chem simulation fails for January start dates in 12.7.0 if SfcVMR true geos-chem#233
• [BUG/ISSUE] Cannot find file with valid time stamp in HEMCO geos-chem#234

Question: How should we handle it? Should we make the fix in geoschem/geos-chem (dev/12.7.2) first and then copy that into HEMCO?

[lizziel]

To be clear, this issue is for the prints not for the timing read bug in HEMCO. I'll put a fix in for this particular print issue in 12.8. A more thorough look at where the prints go, especially testing with GCHP which uses MPI, can be in this repo. I've found that the print behavior can be different between GCHP and GCC.

[lizziel]

An issue related to inconsistent HEMCO error prints messages: GEOS-Chem prints error messages specifying an error in HEMCO with instructions to check HEMCO.log, but starts doing this before HEMCO.log is created. This is confusing and frustrating to users.

[jimmielin]

Also as a side note I'd also like to sneak in a feature for WRF-GC to output HEMCO.log for each core (e.g. HEMCO-0000.log, HEMCO-0001.log, ...). I don't know where the name of the log file HEMCO.log is decided in the code and the file is created, please let me know if you'll change that part of the code and I may follow up with a commit specific to MODEL_WRF 😆

[lizziel]

This update, limited to hco_readlist_mod, will go into 12.8.1. It will be ported to the HEMCO repository as part of the future update for compatibility with 12.8.1.

@jimmielin , regarding outputting HEMCO.log for each core, go for it. Although it would be good if it were optional. You can start a separate GitHub issue for discussion of this.