CMake Build

[islas]

TYPE: new feature

KEYWORDS: CMake, build, make

SOURCE: internal

DESCRIPTION OF CHANGES:
Problem:
The current WRF build system is fragile with many pitfalls making it difficult for users to build & add to it without perpetuating existing problems. Many options exist across various layers of files, languages, and option control-flow.

Solution:
This requires CMake version 3.20 or newer
A redesign of the build system from the ground up, maintaining the interfacing feel and knowledge accumulated in arch/configure.defaults. Condense option selection and control to single locations and as best as possible reduce the complexity of this control.

This will be a work in progress as gaps are identified in reproducing the full functionality of the makefile build system. Currently only em_real and em_ideal have limited supported

Brief how to use:
As this is a work in progress, the original configure and compile scripts have been left as-is. Alongside them are now configure_new and compile_new which walk a user through a slightly similar experience of configuring & compiling WRF.

A simple usage example would be:

# Ensure you have cmake 3.20+ and configuration environment set up
./configure_new
# Follow prompts to select configuration
./compile_new [-j N]

Notable differences are :

• Submodule code must be checked out beforehand and is not checked out during the compile process
• Stanzas presented to a user are only those for which the compiler exists in the current environment
• !! warnings appear for subconfigurations (MPI) that would not be supported in the current environment
• DM/SM selection is now done after selecting a base configuration rather than an individual configuration # within a family of compilers
• Compilation via compile_new does not take target to build as an argument - parallel -j N jobs still supported
• Users do not need to set NETCDF or LD_LIBRARY_PATH variables
• Base binaries do not have .exe extension, but symlinks are provided
• Binaries, test setups, and everything else generated from compilation is copy-placed (not softlinked) to a separate location - default is ./install. This means the equivalent test/em_real/wrf.exe is now at install/test/em_real/wrf.exe

LIST OF MODIFIED FILES:
A CMakeLists.txt
A arch/configure_reader.py
A chem/CMakeLists.txt
A cleanCMake.sh
A cmake/c_preproc.cmake
A cmake/confcheck.cmake
A cmake/gitinfo.cmake
A cmake/m4_preproc.cmake
A cmake/modules/FindJasper.cmake
A cmake/modules/FindRPC.cmake
A cmake/modules/FindnetCDF-Fortran.cmake
A cmake/modules/FindnetCDF.cmake
A cmake/modules/FindpnetCDF.cmake
A cmake/printOption.cmake
A cmake/target_copy.cmake
A cmake/template/WRFConfig.cmake.in
A cmake/template/arch_config.cmake
A cmake/template/commit_decl.cmake
A cmake/wrf_case_setup.cmake
A cmake/wrf_get_version.cmake
A compile_new
A confcheck/CMakeLists.txt
A configure_new
A dyn_em/CMakeLists.txt
A external/CMakeLists.txt
A external/RSL_LITE/CMakeLists.txt
A external/atm_ocn/CMakeLists.txt
A external/esmf_time_f90/CMakeLists.txt
A external/fftpack/fftpack5/CMakeLists.txt
A external/io_adios2/CMakeLists.txt
A external/io_esmf/CMakeLists.txt
A external/io_grib1/CMakeLists.txt
A external/io_grib1/MEL_grib1/CMakeLists.txt
A external/io_grib1/WGRIB/CMakeLists.txt
A external/io_grib1/grib1_util/CMakeLists.txt
A external/io_grib2/CMakeLists.txt
A external/io_grib2/bacio-1.3/CMakeLists.txt
A external/io_grib2/g2lib/CMakeLists.txt
A external/io_grib2/g2lib/utest/CMakeLists.txt
A external/io_grib_share/CMakeLists.txt
A external/io_int/CMakeLists.txt
A external/io_netcdf/CMakeLists.txt
A external/io_netcdfpar/CMakeLists.txt
A external/io_phdf5/CMakeLists.txt
A external/io_pio/CMakeLists.txt
A external/io_pnetcdf/CMakeLists.txt
A external/ioapi_share/CMakeLists.txt
A frame/CMakeLists.txt
A inc/CMakeLists.txt
A main/CMakeLists.txt
A phys/CMakeLists.txt
A share/CMakeLists.txt
A test/em_b_wave/CMakeLists.txt
A test/em_convrad/CMakeLists.txt
A test/em_fire/CMakeLists.txt
A test/em_grav2d_x/CMakeLists.txt
A test/em_heldsuarez/CMakeLists.txt
A test/em_hill2d_x/CMakeLists.txt
A test/em_les/CMakeLists.txt
A test/em_quarter_ss/CMakeLists.txt
A test/em_real/CMakeLists.txt
A test/em_scm_xy/CMakeLists.txt
A test/em_seabreeze2d_x/CMakeLists.txt
A test/em_squall2d_x/CMakeLists.txt
A test/em_squall2d_y/CMakeLists.txt
A test/em_tropical_cyclone/CMakeLists.txt
A tools/CMakeLists.txt
A tools/CodeBase/CMakeLists.txt
A doc/README.cmake_build
M tools/fseek_test.c
M README
M arch/configure.defaults

• Modified file include an adjustment to a compile test to allow the test to be conducted in an out-of-source build manner as prescribed by CMake. Default logic of this test to still test on the existence of Makefile

TESTS CONDUCTED:

1. In various instances this build is faster and more reliable with meaningful diagnostics when errors occur

RELEASE NOTE:
Introduction of a CMake build system for em_real and em_ideal

[mgduda]

I'd like to get some clarification on how the expected workflow under this build system will look for users. Right now, the real and wrf binaries are installed in install/bin with copies in install/test. However, in the install/test directory, although there are binaries and physics tables, example namelists are in a em_real subdirectory. Do we expect users to copy namelists from the install/test/em_real directory into the install/test directory?

Perhaps it's a bit late in the PR process to raise this issue, but we have a large amount of documentation that will need to be updated with the current organization of install and run directories. What if we were to try to mirror the workflow implied by the current build system, but just under the install subdirectory (recognizing that we could support out-of-source builds). So for example, the install/test/em_real directory would contain binaries, physics tables, and namelists -- basically everything in the current test/em_real directory.

Also, is there a strong argument for dropping the .exe suffix from binaries? I don't personally prefer this suffix (as evidenced by the names of executable files in a certain other model), but I do think there's value in trying to minimize unnecessary changes at this point in WRF's lifecycle, especially given the untold number of users who may have scripts for running complex WRF workflows, and given the large body of tutorial material and other documentation.

[weiwangncar]

Building with the HEAD of this PR branch (983f82a) does succeed with the GNU compilers on Derecho for me, so it would appear that the nTiedtke error is related to recent changes in the develop branch.

Could this be related to the newly added physics_mmm/ directory and several physics modules now residing in that directory?

[weiwangncar]

@mgduda @islas From a user's perspective, I'd agree with Michael's suggestion. It would be great if the final executables are named the same as before, perhaps even linked to the current run/ and test/em_real/ (or any of the test/em_ case directory.

[islas]

I'd like to get some clarification on how the expected workflow under this build system will look for users. Right now, the real and wrf binaries are installed in install/bin with copies in install/test. However, in the install/test directory, although there are binaries and physics tables, example namelists are in a em_real subdirectory. Do we expect users to copy namelists from the install/test/em_real directory into the install/test directory?

This is a bug, everything was meant to be installed under test/em_real. In shuffling things around it seems that the binaries and tables from main/ are getting installed one directory higher than they ought to. I'll fix this as soon as I merge develop into this branch - currently running into weird unforeseen issues.

Also, is there a strong argument for dropping the .exe suffix from binaries? I don't personally prefer this suffix (as evidenced by the names of executable files in a certain other model), but I do think there's value in trying to minimize unnecessary changes at this point in WRF's lifecycle, especially given the untold number of users who may have scripts for running complex WRF workflows, and given the large body of tutorial material and other documentation.

It's standard practice for *nix binaries to not have an extension, more so the .exe suffix is most generally applied to Windows binaries. If the workflow can more or less be replicated in the out-of-source build I think this is a much smaller change that could be tolerated.

perhaps even linked to the current run/ and test/em_real/ (or any of the test/em_ case directory

With fixing the install location of tables & binaries we shouldn't need to link back into run/ or test/. The workflow should ideally be going into the out-of-source build (install/) and then from there everything should be the same - though right now a run/ directory does not exist as I was under the impression that almost all runs are started from test/.

[dudhia]

My vote would be to 1. have executables linked to the existing run and test directories where the data and namelist files reside. 2. at least have the .exe suffix in the linked files in those directories to reduce user confusion. I prefer suffixes if only to distinguish files from directories. Jimy

…

On Mon, Feb 12, 2024 at 8:13 PM Anthony Islas ***@***.***> wrote: I'd like to get some clarification on how the expected workflow under this build system will look for users. Right now, the real and wrf binaries are installed in install/bin with copies in install/test. However, in the install/test directory, although there are binaries and physics tables, example namelists are in a em_real subdirectory. Do we expect users to copy namelists from the install/test/em_real directory into the install/test directory? This is a bug, everything was meant to be installed under test/em_real. In shuffling things around it seems that the binaries and tables from main/ are getting installed one directory higher than they ought to. I'll fix this as soon as I merge develop into this branch - currently running into weird unforeseen issues. Also, is there a strong argument for dropping the .exe suffix from binaries? I don't personally prefer this suffix (as evidenced by the names of executable files in a certain other model), but I do think there's value in trying to minimize unnecessary changes at this point in WRF's lifecycle, especially given the untold number of users who may have scripts for running complex WRF workflows, and given the large body of tutorial material and other documentation. It's standard practice for *nix binaries to not have an extension, more so the .exe suffix is most generally applied to Windows binaries. If the workflow can more or less be replicated in the out-of-source build I think this is a much smaller change that could be tolerated. perhaps even linked to the current run/ and test/em_real/ (or any of the test/em_ case directory With fixing the install location of tables & binaries we shouldn't need to link back into run/ or test/. The workflow should ideally be going into the out-of-source build (install/) and then from there everything should be the same - though right now a run/ directory does not exist as I was under the impression that almost all runs are started from test/. — Reply to this email directly, view it on GitHub <#1896 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AEIZ77F6UUUOG5RL32R2C2TYTLK6PAVCNFSM6AAAAAA2CNSMTGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSNBQGM2TMMBTGM> . You are receiving this because your review was requested.Message ID: ***@***.***>

[dudhia]

For me the default intel compiler was successful with qcmd, but I still have problems with configuring when I switch to gcc. It hangs after this. Not sure what the loadedmodules message means. -------------------- Default [0] : [DM] Use MPI? Default [Y] [Y/n] : [SM] Use OpenMP? Default [N] [y/N] : Configure additional options? Default [N] [y/N] : -- Cray Programming Environment 2.7.23 C -- NOTE: LOADEDMODULES changed since initial config! -- NOTE: this may cause unexpected build errors. -- Cray Programming Environment 2.7.23 CXX -- Cray Programming Environment 2.7.23 Fortran

…

On Tue, Feb 13, 2024 at 9:10 AM Jimy Dudhia ***@***.***> wrote: My vote would be to 1. have executables linked to the existing run and test directories where the data and namelist files reside. 2. at least have the .exe suffix in the linked files in those directories to reduce user confusion. I prefer suffixes if only to distinguish files from directories. Jimy On Mon, Feb 12, 2024 at 8:13 PM Anthony Islas ***@***.***> wrote: > I'd like to get some clarification on how the expected workflow under > this build system will look for users. Right now, the real and wrf > binaries are installed in install/bin with copies in install/test. > However, in the install/test directory, although there are binaries and > physics tables, example namelists are in a em_real subdirectory. Do we > expect users to copy namelists from the install/test/em_real directory > into the install/test directory? > > This is a bug, everything was meant to be installed under test/em_real. > In shuffling things around it seems that the binaries and tables from main/ > are getting installed one directory higher than they ought to. I'll fix > this as soon as I merge develop into this branch - currently running into > weird unforeseen issues. > > Also, is there a strong argument for dropping the .exe suffix from > binaries? I don't personally prefer this suffix (as evidenced by the names > of executable files in a certain other model), but I do think there's value > in trying to minimize unnecessary changes at this point in WRF's lifecycle, > especially given the untold number of users who may have scripts for > running complex WRF workflows, and given the large body of tutorial > material and other documentation. > > It's standard practice for *nix binaries to not have an extension, more > so the .exe suffix is most generally applied to Windows binaries. If the > workflow can more or less be replicated in the out-of-source build I think > this is a much smaller change that could be tolerated. > > perhaps even linked to the current run/ and test/em_real/ (or any of the > test/em_ case directory > > With fixing the install location of tables & binaries we shouldn't need > to link back into run/ or test/. The workflow should ideally be going into > the out-of-source build (install/) and then from there everything should be > the same - though right now a run/ directory does not exist as I was under > the impression that almost all runs are started from test/. > > — > Reply to this email directly, view it on GitHub > <#1896 (comment)>, or > unsubscribe > <https://github.com/notifications/unsubscribe-auth/AEIZ77F6UUUOG5RL32R2C2TYTLK6PAVCNFSM6AAAAAA2CNSMTGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSNBQGM2TMMBTGM> > . > You are receiving this because your review was requested.Message ID: > ***@***.***> >

[islas]

It hangs after this. Not sure what the loadedmodules message means.
Default [0] :
[DM] Use MPI? Default [Y] [Y/n] :
[SM] Use OpenMP? Default [N] [y/N] :
Configure additional options? Default [N] [y/N] :
-- Cray Programming Environment 2.7.23 C
-- NOTE: LOADEDMODULES changed since initial config!
-- NOTE: this may cause unexpected build errors.

While I'm not exactly certain what is under the hood causing this the issue arises from swap/switching modules and not clearing the previous build entirely. ./cleanCMake.sh on its own only does make clean, so if this is done then a module is changed, the previous configuration is being used for compilation checks causing cmake to hang indefinitely.

Solution: Do ./cleanCMake.sh -a when switching modules as that invalidates the previous configuration.

[dudhia]

Solution: Do ./cleanCMake.sh -a when switching modules as that invalidates the previous configuration.

Update: This worked for configure and compile after switching to gcc.

[dudhia]

Even though gcc compiled there were some errors in copies of 4 files as the target directory looks wrong.

Error copying file (if different) from "/glade/u/home/dudhia/cmake-test/WRF/run/p3_lookupTable_1.dat-5.3-2momI" to "/glade/u/home/dudhia/cmake-test/WRF/install/test//p3_lookupTable_1.dat-5.3-2momI".

[weiwangncar]

@islas I would still recommend removing COAMPS in the list of models to compile. Yes, it is in the list of things to compile in the old make system, but nobody sees it, and it never was a real option (but no one bothered to remove it, true). But in cmake, it shows up as an option for users. It can be confusing.

[mgduda]

I have just two more requests, and then I think any remaining cleanup issues can be ironed out before the release:

1. Right now it looks like namelists are placed in install/test/em_real/em_real, and I think these should go in install/test/em_real.
2. Would it make sense to symlink the binaries from install/bin to the install/test/em_real directory rather than copying them? I would also propose that we have symlinks to the binaries in the install/test/em_real directory with the .exe suffix. I know this isn't the Unix way, but I really think there's value in not forcing unnecessary updates to documentation, workflows, etc. As a compromise, I wouldn't object to having both names in the install/test/em_real directory (with and without the .exe suffix).

[weiwangncar]

@rcabell Can you help review and approve this PR?