[develop] Move all unittest tests to a common area.

[christinaholtNOAA]

DESCRIPTION OF CHANGES:

Moves all unittest tests to the tests directory, and lints them. They currently should pass at 10/10. This makes running them easier so that users may be encouraged and enabled to do that on their own platform.

Adds the HPSS functional tests to the Jenkinsfile, so that the full functionality is tested regularly. These tests run on the front end so shouldn't eat into allocation. The tests clean up after themselves, so shouldn't be a data issue in the long-run.

Adds linting for the tests directory to the GitHub Actions workflow to ensure they stay linted.

Type of change

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to not work as expected)
• This change requires a documentation update

TESTS CONDUCTED:

• hera.intel
• orion.intel
• cheyenne.intel
• cheyenne.gnu
• gaea.intel
• jet.intel
• wcoss2.intel
• NOAA Cloud (indicate which platform)
• Jenkins
• fundamental test suite
• comprehensive tests (specify which if a subset was used)

DEPENDENCIES:

None. Someone should check the Jenkinsfile though! I can't test that, really.

DOCUMENTATION:

Maybe?

ISSUE:

Fixes Issue #726.

CHECKLIST

• My code follows the style guidelines in the Contributor's Guide
• I have performed a self-review of my own code using the Code Reviewer's Guide
• I have commented my code, particularly in hard-to-understand areas
• My changes need updates to the documentation. I have made corresponding changes to the documentation
• My changes do not require updates to the documentation (explain).
• My changes generate no new warnings
• New and existing tests pass with my changes
• Any dependent changes have been merged and published

LABELS (optional):

A Code Manager needs to add the following labels to this PR:

• Work In Progress
• bug
• enhancement
• documentation
• release
• high priority
• run_ci
• run_we2e_fundamental_tests
• run_we2e_comprehensive_tests
• Needs Cheyenne test
• Needs Jet test
• Needs Hera test
• Needs Orion test
• help wanted

[panll]

Tested on Hera, and it works fine

[MichaelLueken]

@christinaholtNOAA I have created a sandbox pipeline in Jenkins for testing PRs with modified Jenkinsfiles. The test failed in the initialization step due to a missing ' at the end of Functional Tests on line 127. I also noted the lack of bash --login and " on line 130, which will likely cause problems as well. Once these changes have been made, I'll attempt to run the Jenkins tests again.

[MichaelLueken]

@christinaholtNOAA Jenkins doesn't like the location of parallel (it expected stage, not parallel). I've provided an example of what the documentation would suggest a parallel build would look like, but I have no idea how to include post. Would it be okay to remove parallel { from line 125 and just have a three-part pipeline?

[MichaelLueken]

@christinaholtNOAA Thank you very much for applying the changes to the Jenkinsfile! Jenkins is now able to run the updated pipeline with the Functional Tests section. However, the Functional Tests section is failing. The failure appears to be due to lines 8 and 9 in srw_unittest.sh. I have added a suggestion that should correct the issue.

[MichaelLueken]

@christinaholtNOAA PR #736 was merged this morning. This updated the test_retrieve_data.py unittest script. This caused a conflict in your PR. Please merge your PR to the latest develop.

[christinaholtNOAA]

@MichaelLueken I resolved the conflicts from last week. Let me know if there are any remaining issues with the Jenkins file. Thanks!

[MichaelLueken]

@christinaholtNOAA Running the new srw_unittest.sh script through the Jenkins pipeline shows a few more issues. I have manually run the script on Hera and have provided feedback to address everything that I encountered.

However, the test is failing to run with the following error:

+ export PYTHONPATH=/scratch2/NAGAPE/epic/Michael.Lueken/srw_unittest/ush
+ PYTHONPATH=/scratch2/NAGAPE/epic/Michael.Lueken/srw_unittest/ush
+ python -m unittest /scratch2/NAGAPE/epic/Michael.Lueken/srw_unittest/tests/test_python/test_retrieve_data.py
E
======================================================================
ERROR: /scratch2/NAGAPE/epic/Michael (unittest.loader._FailedTest)
----------------------------------------------------------------------
ImportError: Failed to import test module: /scratch2/NAGAPE/epic/Michael
Traceback (most recent call last):
  File "/scratch1/NCEPDEV/nems/role.epic/miniconda3/4.12.0/envs/regional_workflow/lib/python3.9/unittest/loader.py", line 154, in loadTestsFromName
    module = __import__(module_name)
ModuleNotFoundError: No module named '/scratch2/NAGAPE/epic/Michael'


----------------------------------------------------------------------
Ran 1 test in 0.000s

Have you seen this behavior before? I'll try to continue digging into this, but I can't tell if there is an issue with the test, the machine, or in the conda environment itself.

[MichaelLueken]

@christinaholtNOAA With the latest changes that you pushed to your branch, I'm now able to manually run the new srw_unittest.sh script. I will now resubmit this PR in the Jenkins pipeline and see how it goes. Thanks for working with me on this work!

[MichaelLueken]

@christinaholtNOAA The new Functional Tests section is running on Hera. This section, however, failed on Jet with the following error:

======================================================================
ERROR: /lfs4/HFIP/hfv3gfs/role (unittest.loader._FailedTest)
----------------------------------------------------------------------
ImportError: Failed to import test module: /lfs4/HFIP/hfv3gfs/role
Traceback (most recent call last):
  File "/mnt/lfs4/HFIP/hfv3gfs/role.epic/miniconda3/4.12.0/envs/regional_workflow/lib/python3.9/unittest/loader.py", line 154, in loadTestsFromName
    module = __import__(module_name)
ModuleNotFoundError: No module named '/lfs4/HFIP/hfv3gfs/role'


----------------------------------------------------------------------

It looks like using:

    cd ${workspace}/tests/test_python
    export PYTHONPATH=${workspace}/ush
    python -m unittest test_retrieve_data.py

should allow the script to run on both Hera and Jet.

Due to the nature of the testing, a failure will stop the entire pipeline, so this will be good to also ensure that everything that we are testing with test_retrieve_data.py is behaving as expected. This also means that the Jet and Hera tests will fail to run through due to HPSS being down until sometime tomorrow.

[christinaholtNOAA]

We shouldn't need to cd to the test area to run this test. I'm concerned about needing to be in a specific sub-path because of any other files we might want to pull in from elsewhere in the repo.

I had a bit of trouble reproducing this failure as my own user, and finally figured out that it seems related to the . in the epic role account path.

I think it may work with a relative path and no cd like this:

  export PYTHONPATH=${workspace}/ush
  python -m unittest .tests/test_python/test_retrieve_data.py

I will push the change to see if that works.

[MichaelLueken]

I was able to manually run the srw_unittest.sh script on Jet following your latest update (23623c4). Once the current tests conclude, I will resubmit the tests to make sure that Jenkins is happy with Jet.

Please note that I would like to run one last test once HPSS has been returned, to ensure that the functional tests pass with HPSS and AWS. Then, we can move forward with merging this work.

[MichaelLueken]

@christinaholtNOAA It looks like the new Functional Tests section are working correctly for both Hera and Jet.

The only issue is that the Functional Tests now take between one and two hours to run on Hera due to the inclusion from PR #736. The test is pulling several large NEMSIO files, significantly increasing run times for the two ufs-case-study pulls from AWS. Please note that this only appears to be an issue on Hera. The Jet test successfully passes in 17 minutes and the GitHub Actions test passes in 12 minutes. Issue #753 was opened detailing the work to correct this.

As part of this PR, it might be a good idea to turn off the two ufs-case-study tests until the ICs/LBCs have been replaced by grib2 data.

[MichaelLueken]

On Cheyenne GNU, the nco_grid_RRFS_CONUS_3km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v15_thompson_mynn_lam3km test failed. This test failed in run_post, which has been happening since this test has been added and is likely due to issue #652.

On Cheyenne Hera, the grid_RRFS_CONUS_25km_ics_NAM_lbcs_NAM_suite_GFS_v16 test failed. This test failed in run_MET_EnsembleStat_vx_RETOP, which has been happening for a long time now and is likely due to issue #688.

On Hera GNU, the get_from_HPSS_ics_FV3GFS_lbcs_FV3GFS_fmt_nemsio_2019061200 test failed in get_extrn_lbcs_201906120000, the get_from_NOMADS_ics_FV3GFS_lbcs_FV3GFS test failed in get_extrn_lbcs_202305030000 - which is an issue that has been happening since the test was added, and the MET_verification_only_vx test failed in run_MET_GridStat_vx_REFC_mem000_20190615. I will be rerunning the get_from_HPSS_ics_FV3GFS_lbcs_FV3GFS_fmt_nemsio_2019061200 and MET_verification_only_vx tests to make sure that they run.

On Hera Intel, the grid_RRFS_CONUS_25km_ics_FV3GFS_lbcs_RAP_suite_HRRR test failed in run_MET_PcpCombine_fcst_APCP01h_mem000, as documented in issue #688.

After rerunning the two failed tests on Hera GNU, so long as they pass, I will move forward with merging this work.

[MichaelLueken]

Rerunning the get_from_HPSS_ics_FV3GFS_lbcs_FV3GFS_fmt_nemsio_2019061200 and MET_verification_only_vx tests showed that they successfully completed. Will now move forward with merging this PR to develop.