Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Testing ESMF 8.6.0 in ufs weather model #2036

Closed
junwang-noaa opened this issue Dec 14, 2023 · 36 comments · Fixed by #2302
Closed

Testing ESMF 8.6.0 in ufs weather model #2036

junwang-noaa opened this issue Dec 14, 2023 · 36 comments · Fixed by #2302
Assignees
@junwang-noaa @BrianCurtis-NOAA
Labels
enhancement New feature or request

Comments

@junwang-noaa
Copy link
Collaborator

Description

ESMF 8.6.0 is released, it needs to be tested in ufs-weather-model.

Solution

Alternatives

Related to
@junwang-noaa junwang-noaa added the enhancement New feature or request label Dec 14, 2023
@BrianCurtis-NOAA BrianCurtis-NOAA mentioned this issue Jan 8, 2024
Merged
42 tasks
@junwang-noaa
Copy link
Collaborator Author

Test ESMF 8.6.0 on acorn in spack-stack 1.6.0. All tests passed with no results change.
Hang installed it in hpc-stack, will test it when the module file is ready.

@uturuncoglu uturuncoglu mentioned this issue Jan 18, 2024
Open
@junwang-noaa
Copy link
Collaborator Author

From Hang:

the spack-stack 1.6.0 has been installed on acorn under
/lfs/h1/emc/nceplibs/noscrub/spack-stack/ This should work fine with ufs
default modulefiles for spack-stack. For the running with hpc-stack
installations, please use following modulefile.

A exactly same library settings to the UFS repo was made on acorn for
further testing:
/lfs/h1/emc/eib/noscrub/Hang.Lei/ufs-weather-model/modulefiles/ufs_acorn.intel.lua
/lfs/h1/emc/eib/noscrub/Hang.Lei/ufs-weather-model/modulefiles/ufs_common.lua

We are still missing some latest versions of libraries on cactus and
Dogwoods to match spack-stack. I will test a wcoss2 version based on
current libraries on Cactus. And push missing versions there soon.

@BrianCurtis-NOAA
Copy link
Collaborator

@Hang-Lei-NOAA I cannot use the provided modulefiles to test 8.6.0 using the hpc-stack. There is an issue with the script finding bacio or netcdf in some cases. I've tried permutations of it with no success either. Please provide a working modulefile for the hpc-stack setup to test ESMF 8.6.0.

@Hang-Lei-NOAA
Copy link

Hang-Lei-NOAA commented Feb 23, 2024 via email

@BrianCurtis-NOAA
Copy link
Collaborator

@Hang-Lei-NOAA @junwang-noaa Currently, if I use that modulesfiles on the top of develop branch, in my case I just tried building the control_c48 case and it fails at finding NetCDF. Is there a CMakeModules I'm supposed to update to as well?

@junwang-noaa
Copy link
Collaborator Author

I don't think so. @Hang-Lei-NOAA would you please check the ESMF/8.6.0 module path?

@Hang-Lei-NOAA
Copy link

Hang-Lei-NOAA commented Mar 4, 2024 via email

@BrianCurtis-NOAA
Copy link
Collaborator

@Hang-Lei-NOAA my issue is with netcdf. CMake is not finding NetCDF.

@Hang-Lei-NOAA
Copy link

Hang-Lei-NOAA commented Mar 4, 2024 via email

@BrianCurtis-NOAA
Copy link
Collaborator

It is the same ones you linked to me:

/lfs/h1/emc/eib/noscrub/Hang.Lei/ufs-weather-model/modulefiles/ufs_acorn.intel.lua
/lfs/h1/emc/eib/noscrub/Hang.Lei/ufs-weather-model/modulefiles/ufs_common.lua

I've only modified the ESMF & MAPL version to 8.6.0

@junwang-noaa
Copy link
Collaborator Author

@BrianCurtis-NOAA what's the error message?

@BrianCurtis-NOAA
Copy link
Collaborator

@junwang-noaa @Hang-Lei-NOAA

CMAKE_FLAGS="-DAPP=S2S -DCCPP_SUITES=FV3_GFS_v17_coupled_p8" ./build.sh 
-- The C compiler identification is Intel 19.1.3.20200925
-- The CXX compiler identification is Intel 19.1.3.20200925
-- The Fortran compiler identification is Intel 19.1.3.20200925
-- Cray Programming Environment 2.7.13 C
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /opt/cray/pe/craype/2.7.13/bin/cc - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Cray Programming Environment 2.7.13 CXX
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /opt/cray/pe/craype/2.7.13/bin/CC - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Cray Programming Environment 2.7.13 Fortran
-- Detecting Fortran compiler ABI info
-- Detecting Fortran compiler ABI info - done
-- Check for working Fortran compiler: /opt/cray/pe/craype/2.7.13/bin/ftn - skipped
-- Checking whether /opt/cray/pe/craype/2.7.13/bin/ftn supports Fortran 90
-- Checking whether /opt/cray/pe/craype/2.7.13/bin/ftn supports Fortran 90 - yes

Configuring UFS app in S2S mode

FMS .............. ON
FV3 .............. ON
AQM .............. OFF
GOCART ........... OFF
MOM6 ............. ON
HYCOM ............ OFF
CICE6 ............ ON
WW3 .............. OFF
STOCH_PHYS ....... ON
CDEPS ............ OFF
CMEPS ............ ON
NOAHMP ........... OFF

Setting configuration for acorn

32BIT ............ OFF
CCPP_32BIT ....... OFF
AVX2 ............. OFF
SIMDMULTIARCH .... OFF
DEBUG ............ OFF
INLINE_POST ...... ON
MULTI_GASES ...... OFF
MOVING_NEST ...... OFF
REGIONAL_MOM6..... OFF
OPENMP ........... ON
PARALLEL_NETCDF .. ON
JEDI_DRIVER ...... OFF
CMEPS_AOFLUX ..... OFF
CDEPS_INLINE ..... OFF

-- Found MPI_C: /opt/cray/pe/craype/2.7.13/bin/cc (found version "3.1") 
-- Found MPI_CXX: /opt/cray/pe/craype/2.7.13/bin/CC (found version "3.1") 
-- Found MPI_Fortran: /opt/cray/pe/craype/2.7.13/bin/ftn (found version "3.1") 
-- Found MPI: TRUE (found version "3.1")  
-- Found OpenMP_C: -qopenmp (found version "5.0") 
-- Found OpenMP_CXX: -qopenmp (found version "5.0") 
-- Found OpenMP_Fortran: -qopenmp (found version "5.0") 
-- Found OpenMP: TRUE (found version "5.0")  
CMake Error at CMakeModules/Modules/FindNetCDF.cmake:246 (message):
  Unable to properly find NetCDF.  Found static libraries at:
  /lfs/h1/emc/nems/noscrub/brian.curtis/git/BrianCurtis-NOAA/ufs-weather-model/esmf860/NetCDF_C_LIBRARY-NOTFOUND
  but could not run nc-config:
Call Stack (most recent call first):
  CMakeLists.txt:147 (find_package)


CMake Error at CMakeModules/Modules/FindNetCDF.cmake:246 (message):
  Unable to properly find NetCDF.  Found static libraries at:
  /lfs/h1/emc/nems/noscrub/brian.curtis/git/BrianCurtis-NOAA/ufs-weather-model/esmf860/NetCDF_Fortran_LIBRARY-NOTFOUND
  but could not run nc-config:
Call Stack (most recent call first):
  CMakeLists.txt:147 (find_package)


CMake Error at /apps/spack/cmake/3.20.2/intel/19.1.3.304/a43g47cjru4jlyziud2y2zc7k6jnbsn2/share/cmake-3.20/Modules/FindPackageHandleStandardArgs.cmake:230 (message):
  Could NOT find NetCDF (missing: NetCDF_INCLUDE_DIRS NetCDF_LIBRARIES C
  Fortran) (Required is at least version "4.7.4")
Call Stack (most recent call first):
  /apps/spack/cmake/3.20.2/intel/19.1.3.304/a43g47cjru4jlyziud2y2zc7k6jnbsn2/share/cmake-3.20/Modules/FindPackageHandleStandardArgs.cmake:594 (_FPHSA_FAILURE_MESSAGE)
  CMakeModules/Modules/FindNetCDF.cmake:312 (find_package_handle_standard_args)
  CMakeLists.txt:147 (find_package)


-- Configuring incomplete, errors occurred!
See also "/lfs/h1/emc/nems/noscrub/brian.curtis/git/BrianCurtis-NOAA/ufs-weather-model/esmf860/build/CMakeFiles/CMakeOutput.log".

@Hang-Lei-NOAA
Copy link

Hang-Lei-NOAA commented Mar 4, 2024 via email

@BrianCurtis-NOAA
Copy link
Collaborator

@Hang-Lei-NOAA would you be able to use https://github.com/ufs-community/ufs-weather-model/blob/develop/modulefiles/ufs_wcoss2.intel.lua for your ufs_acorn.intel.lua (this skips the need for ufs_common.lua which wcoss2 will never use.

@Hang-Lei-NOAA
Copy link

Hang-Lei-NOAA commented Mar 4, 2024 via email

@BrianCurtis-NOAA
Copy link
Collaborator

@Hang-Lei-NOAA using your ufs_acorn.intel.lua and ufs_common.lua i get the following error in CMake:

CMAKE_FLAGS="-DAPP=S2S -DCCPP_SUITES=FV3_GFS_v17_coupled_p8" ./build.sh 
-- The C compiler identification is Intel 19.1.3.20200925
-- The CXX compiler identification is Intel 19.1.3.20200925
-- The Fortran compiler identification is Intel 19.1.3.20200925
-- Cray Programming Environment 2.7.13 C
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /opt/cray/pe/craype/2.7.13/bin/cc - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Cray Programming Environment 2.7.13 CXX
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /opt/cray/pe/craype/2.7.13/bin/CC - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Cray Programming Environment 2.7.13 Fortran
-- Detecting Fortran compiler ABI info
-- Detecting Fortran compiler ABI info - done
-- Check for working Fortran compiler: /opt/cray/pe/craype/2.7.13/bin/ftn - skipped
-- Checking whether /opt/cray/pe/craype/2.7.13/bin/ftn supports Fortran 90
-- Checking whether /opt/cray/pe/craype/2.7.13/bin/ftn supports Fortran 90 - yes

Configuring UFS app in S2S mode

FMS .............. ON
FV3 .............. ON
AQM .............. OFF
GOCART ........... OFF
MOM6 ............. ON
HYCOM ............ OFF
CICE6 ............ ON
WW3 .............. OFF
STOCH_PHYS ....... ON
CDEPS ............ OFF
CMEPS ............ ON
NOAHMP ........... OFF

Setting configuration for acorn

32BIT ............ OFF
CCPP_32BIT ....... OFF
AVX2 ............. OFF
SIMDMULTIARCH .... OFF
DEBUG ............ OFF
INLINE_POST ...... ON
MULTI_GASES ...... OFF
MOVING_NEST ...... OFF
REGIONAL_MOM6..... OFF
OPENMP ........... ON
PARALLEL_NETCDF .. ON
JEDI_DRIVER ...... OFF
CMEPS_AOFLUX ..... OFF
CDEPS_INLINE ..... OFF

-- Found MPI_C: /opt/cray/pe/craype/2.7.13/bin/cc (found version "3.1") 
-- Found MPI_CXX: /opt/cray/pe/craype/2.7.13/bin/CC (found version "3.1") 
-- Found MPI_Fortran: /opt/cray/pe/craype/2.7.13/bin/ftn (found version "3.1") 
-- Found MPI: TRUE (found version "3.1")  
-- Found OpenMP_C: -qopenmp (found version "5.0") 
-- Found OpenMP_CXX: -qopenmp (found version "5.0") 
-- Found OpenMP_Fortran: -qopenmp (found version "5.0") 
-- Found OpenMP: TRUE (found version "5.0")  
-- Found NetCDF: /apps/prod/hpc-stack/i-19.1.3.304__m-8.1.12__h-1.14.0__n-4.9.2__p-2.5.10__e-8.4.2/intel-19.1.3.304/cray-mpich-8.1.12/netcdf/4.9.2/include (found suitable version "4.9.2", minimum required is "4.7.4") found components: C Fortran 
-- FindNetCDF defines targets:
--   - NetCDF_VERSION [4.9.2]
--   - NetCDF_PARALLEL [TRUE]
--   - NetCDF_C_CONFIG_EXECUTABLE [/apps/prod/hpc-stack/i-19.1.3.304__m-8.1.12__h-1.14.0__n-4.9.2__p-2.5.10__e-8.4.2/intel-19.1.3.304/cray-mpich-8.1.12/netcdf/4.9.2/bin/nc-config]
--   - NetCDF::NetCDF_C [SHARED] [Root: /apps/prod/hpc-stack/i-19.1.3.304__m-8.1.12__h-1.14.0__n-4.9.2__p-2.5.10__e-8.4.2/intel-19.1.3.304/cray-mpich-8.1.12/netcdf/4.9.2] Lib: /apps/prod/hpc-stack/i-19.1.3.304__m-8.1.12__h-1.14.0__n-4.9.2__p-2.5.10__e-8.4.2/intel-19.1.3.304/cray-mpich-8.1.12/netcdf/4.9.2/lib/libnetcdf.so 
--   - NetCDF_Fortran_CONFIG_EXECUTABLE [/apps/prod/hpc-stack/i-19.1.3.304__m-8.1.12__h-1.14.0__n-4.9.2__p-2.5.10__e-8.4.2/intel-19.1.3.304/cray-mpich-8.1.12/netcdf/4.9.2/bin/nf-config]
--   - NetCDF::NetCDF_Fortran [SHARED] [Root: /apps/prod/hpc-stack/i-19.1.3.304__m-8.1.12__h-1.14.0__n-4.9.2__p-2.5.10__e-8.4.2/intel-19.1.3.304/cray-mpich-8.1.12/netcdf/4.9.2] Lib: /apps/prod/hpc-stack/i-19.1.3.304__m-8.1.12__h-1.14.0__n-4.9.2__p-2.5.10__e-8.4.2/intel-19.1.3.304/cray-mpich-8.1.12/netcdf/4.9.2/lib/libnetcdff.so 
-- Found ESMF: /lfs/h1/emc/nceplibs/noscrub/hpc-stack/libs/hpc-stack/intel-19.1.3.304/cray-mpich-8.1.9/esmf/8.6.0/lib/libesmf.a (found suitable version "8.6.0", minimum required is "8.3.0") 
-- Found MPI: TRUE (found version "3.1") found components: Fortran 
-- Found OpenMP_Fortran: -qopenmp (found version "5.0") 
-- Found OpenMP: TRUE (found version "5.0") found components: Fortran 
-- Found MPI: TRUE (found version "3.1")  
-- Found FMS: "/lfs/h1/emc/nceplibs/noscrub/hpc-stack/libs/hpc-stack/intel-19.1.3.304/cray-mpich-8.1.9/fms/2023.04" (Version: "2023.4.0")
-- FMS targets:
--   - FMS::fms_r4 [Lib: /lfs/h1/emc/nceplibs/noscrub/hpc-stack/libs/hpc-stack/intel-19.1.3.304/cray-mpich-8.1.9/fms/2023.04/lib/libfms_r4.a]
--   - FMS::fms_r8 [Lib: /lfs/h1/emc/nceplibs/noscrub/hpc-stack/libs/hpc-stack/intel-19.1.3.304/cray-mpich-8.1.9/fms/2023.04/lib/libfms_r8.a]
-- Found PIO: /apps/prod/intel-19.1.3.304/cray-mpich-8.1.9/pio/2.5.10 (found suitable version "2.5.10", minimum required is "2.5.3") found components: C Fortran 
-- FindPIO:
--   - PIO_PREFIX [/apps/prod/intel-19.1.3.304/cray-mpich-8.1.9/pio/2.5.10]
--   - PIO_VERSION: [2.5.10]
--   - PIO Components Found: C;Fortran
-- Found bacio: /apps/ops/prod/libs/intel/19.1.3.304/bacio/2.4.1/lib/libbacio_4.a (found version "2.4.0")
-- Found OpenMP_Fortran: -qopenmp (found version "5.0") 
-- Found sp: /apps/ops/prod/libs/intel/19.1.3.304/sp/2.3.3/lib/libsp_4.a (found version "2.3.3")
CMake Error at CMakeLists.txt:171 (find_package):
  By not providing "Findw3emc.cmake" in CMAKE_MODULE_PATH this project has
  asked CMake to find a package configuration file provided by "w3emc", but
  CMake did not find one.

  Could not find a package configuration file provided by "w3emc" (requested
  version 2.9.2) with any of the following names:

    w3emcConfig.cmake
    w3emc-config.cmake

  Add the installation prefix of "w3emc" to CMAKE_PREFIX_PATH or set
  "w3emc_DIR" to a directory containing one of the above files.  If "w3emc"
  provides a separate development package or SDK, be sure it has been
  installed.


-- Configuring incomplete, errors occurred!
See also "/lfs/h1/emc/nems/noscrub/brian.curtis/git/BrianCurtis-NOAA/ufs-weather-model/esmf860/build/CMakeFiles/CMakeOutput.log".

@Hang-Lei-NOAA
Copy link

Hang-Lei-NOAA commented Mar 5, 2024 via email

@BrianCurtis-NOAA
Copy link
Collaborator

@Hang-Lei-NOAA

CMake Error at /apps/spack/cmake/3.20.2/intel/19.1.3.304/a43g47cjru4jlyziud2y2zc7k6jnbsn2/share/cmake-3.20/Modules/CMakeFindDependencyMacro.cmake:47 (find_package):
  By not providing "Findbufr.cmake" in CMAKE_MODULE_PATH this project has
  asked CMake to find a package configuration file provided by "bufr", but
  CMake did not find one.

  Could not find a package configuration file provided by "bufr" with any of
  the following names:

    bufrConfig.cmake
    bufr-config.cmake

  Add the installation prefix of "bufr" to CMAKE_PREFIX_PATH or set
  "bufr_DIR" to a directory containing one of the above files.  If "bufr"
  provides a separate development package or SDK, be sure it has been
  installed.
Call Stack (most recent call first):
  /lfs/h1/emc/nceplibs/noscrub/hpc-stack/libs/hpc-stack/intel-19.1.3.304/w3emc/2.10.0/lib64/cmake/w3emc/w3emc-config.cmake:46 (find_dependency)
  CMakeLists.txt:171 (find_package)


-- Configuring incomplete, errors occurred!
See also "/lfs/h1/emc/nems/noscrub/brian.curtis/git/BrianCurtis-NOAA/ufs-weather-model/esmf860/build/CMakeFiles/CMakeOutput.log".

@Hang-Lei-NOAA
Copy link

Hang-Lei-NOAA commented Mar 5, 2024 via email

@Hang-Lei-NOAA
Copy link

Hang-Lei-NOAA commented Mar 5, 2024 via email

@BrianCurtis-NOAA
Copy link
Collaborator

Something is wrong with that hdf5:

/var/run/palsd/61fdedd2-98c0-485f-addd-4e5a51d5bcdf/files/fv3.exe: error while loading shared libraries: libhdf5_hl.so.310: cannot open shared object file: No such file or directory
/var/run/palsd/61fdedd2-98c0-485f-addd-4e5a51d5bcdf/files/fv3.exe: error while loading shared libraries: libhdf5_hl.so.310: cannot open shared object file: No such file or directory
/var/run/palsd/61fdedd2-98c0-485f-addd-4e5a51d5bcdf/files/fv3.exe: error while loading shared libraries: libhdf5_hl.so.310: cannot open shared object file: No such file or directory

@Hang-Lei-NOAA
Copy link

Hang-Lei-NOAA commented Mar 6, 2024 via email

@Hang-Lei-NOAA
Copy link

Hang-Lei-NOAA commented Mar 6, 2024 via email

@Hang-Lei-NOAA
Copy link

Hang-Lei-NOAA commented Mar 7, 2024 via email

@BrianCurtis-NOAA
Copy link
Collaborator

@Hang-Lei-NOAA on WCOSS2 I get issues with netcdff finding the appropriate so.

./fv3.exe: error while loading shared libraries: libnetcdff.so.7: cannot open shared object file: No such file or directory

@Hang-Lei-NOAA
Copy link

Hang-Lei-NOAA commented Mar 7, 2024 via email

@Hang-Lei-NOAA
Copy link

Hang-Lei-NOAA commented Mar 7, 2024 via email

@BrianCurtis-NOAA
Copy link
Collaborator

BrianCurtis-NOAA commented Mar 7, 2024 •
edited
Loading

@Hang-Lei-NOAA
run_dir: /lfs/h2/emc/ptmp/brian.curtis/FV3_RT/rt_171470/regional_atmaq_debug_intel

my setup is: /lfs/h2/emc/nems/noscrub/brian.curtis/git/BrianCurtis-NOAA/ufs-weather-model/develop/

@Hang-Lei-NOAA
Copy link

Hang-Lei-NOAA commented Mar 7, 2024 via email

@Hang-Lei-NOAA
Copy link

WCOSS2 Ticket number 2024031510000019

Assigned to Bongi

Pushed on April/4

@Hang-Lei-NOAA
Copy link

Associated mapl/2.40.3-esmf-8.6.0 version updates is with Ticket number 2024031510000028
Assigned to Bongi
Pushed on April/4

@junwang-noaa
Copy link
Collaborator Author

@Hang-Lei-NOAA Do you hear anything from Bongi? Are the libraries available on wcoss2 now?

@Hang-Lei-NOAA
Copy link

@junwang-noaa after we tested on acron, it was Tuesday. GDIT then need to ask NCO to start the RFC ticket. At least it will take a week to get it on wcoss2. Please just wait. I will inform Brian once it is ready for confirm testing.

@Hang-Lei-NOAA
Copy link

Lib-C series are available on cactus for quick fully testing. @junwang-noaa

@junwang-noaa
Copy link
Collaborator Author

@BrianCurtis-NOAA just to confirm that you will test the ESMF 8.6.0 in your test with other library update (pio netcdf), right?

@BrianCurtis-NOAA
Copy link
Collaborator

Yes

@BrianCurtis-NOAA BrianCurtis-NOAA mentioned this issue May 30, 2024
Merged
14 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@junwang-noaa junwang-noaa

@BrianCurtis-NOAA BrianCurtis-NOAA

Labels
enhancement New feature or request
Projects
No open projects
Model infrastructure development FY2024Q2
Status: In Progress
Model infrastructure development FY2024Q3
Status: Done
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Update WCOSS2 libraries for spack-stack 1.6.0 (Combined PR #2301) BrianCurtis-NOAA/ufs-weather-model
3 participants
@junwang-noaa @Hang-Lei-NOAA @BrianCurtis-NOAA