Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Feature/rt automation #403

Merged
merged 159 commits into from
Feb 25, 2021
Merged

Feature/rt automation #403

merged 159 commits into from
Feb 25, 2021

Conversation

BrianCurtis-NOAA
Copy link
Collaborator

@BrianCurtis-NOAA BrianCurtis-NOAA commented Feb 1, 2021 •
edited
Loading

Description

Automation of Regression Testing

This PR adds the directory auto to tests that contains three files rt_auto.yml, rt_auto.sh and rt_auto.py.

Code managers will add labels to PR's that are ready for regression testing and eventually a CRON job will run this script every 20 minutes.

  1. rt_auto.py: Python code to be used on our supported HPC's to automatically check all pull requests to a specific repository for a specific label. Each label will contain three parts, for example a label is named Auto-RT-hera
    A: Auto which tells the python code that label is to be processed.
    B: RT which is the test requested to run (check rt_auto.yml for test labels).
    C: hera which specifies the machine the test is to be run on.
    Currently Auto-RT-hera Auto-RT-orion Auto-RT-gaea are working. I'm looking to get python packages added on Jet first.
  2. rt_auto.yml: Specific information the code needs to run
  3. rt_auto.sh: a bash script to run the python code, soon to be implemented through cron jobs.

Current functions:

  1. Automatically clones the PR repo
  2. Automatically runs the full regression test suite
  3. Commits/Pushes logs to the PR repo
  4. Deletes the cloned repo it created to run tests (NOTE: Does NOT yet delete the stmp run data automatically)
  5. Each run, checks for closed PR's and deleted the directory it had created to run those regression tests. (NOTE: Also does not delete the stmp run dir)

Planned future functions:

  • Automatically delete the STMP run dir
  • Automatically create new baselines
  • Automatically run full regression test suite on new baselines
  • Automatically add new baselines to baseline directory

Discussion Topics:

  • Currently the planned CRON job implementation will use BrianCurtis-NOAA HPC accounts and GitHub credentials to run. This optimally should move to a bot GitHub account and a generic user account on NOAA HPC's.

Issue(s) addressed

Fixes #306

Testing

The use of this new script will be used as its test.

Dependencies

None

Do PRs in upstream repositories need to be merged first?
No, these changes are independent of the main functionality of the repo.

BuildTools and others added 30 commits November 23, 2020 08:45
rt automation script
54b1826
rt automation script w/o token
9d93b8d
getting machine name test
f8dfa9f
Updates to python script, and addition of YAML database
cff8446
added check for approved users, added check for triggers in pr comments
89c6248
Script now interacts with github directly, is able to process triggers
d35e286
Merge branch 'develop' into feature/rt-automation
ad2364f
Added repo clone and submodule functions
ed3450f
Merge branch 'develop' into feature/rt-automation
0b3a691
Added threaded process submition with callback for regression tests
84154fe
Merge branch 'develop' into feature/rt-automation
ab8c83e
@BrianCurtis-NOAA
remove an extra sys.exit()
2ba8ec2
@BrianCurtis-NOAA
Switched to using Labels
cc57e45
@BrianCurtis-NOAA
Merge branch 'develop' into feature/rt-automation
a5ecb16
@BrianCurtis-NOAA
Rename RegressionTests_hera.intel.log to RegressionTests_neon.intel.log
3db3ff2 
temp change to test function
@BrianCurtis-NOAA
Rename RegressionTests_neon.intel.log to RegressionTests_hera.intel.log
ce666c5 
reverted
@BrianCurtis-NOAA
AUTO: Updated tests log
c588de7
@BrianCurtis-NOAA
Test change for repo
31f10e5
@BrianCurtis-NOAA
Merge branch 'feature/rt-automation' of https://github.com/BrianCurti…
3bbef07 
…s-NOAA/ufs-weather-model into feature/rt-automation
@BrianCurtis-NOAA
Revert changes for testing
239d839
@BrianCurtis-NOAA
Auto: Added Updated RT Log file: tests/RegressionTests_hera.intel.log
923c46d
@BrianCurtis-NOAA
Auto: Added Updated RT Log file: tests/RegressionTests_hera.intel.log
94d1e73
@BrianCurtis-NOAA
Auto: Added Updated RT Log file: tests/RegressionTests_hera.intel.log
61aae99
@BrianCurtis-NOAA
Auto: Added Updated RT Log file: tests/RegressionTests_hera.intel.log
e3c2931
@BrianCurtis-NOAA
Add's ability to retrieve RT logs for PR repo. Moves access token ext…
f47c1ad 
…ernal to code
@BrianCurtis-NOAA
Merge branch 'feature/rt-automation' of https://github.com/BrianCurti…
1b7c5f4 
…s-NOAA/ufs-weather-model into feature/rt-automation
@BrianCurtis-NOAA
TEST ADDITION HOPEFUL
7ffde33 
L
@BrianCurtis-NOAA
Almost done implementing RT log files back to PR, saving files for he…
e32eb4b 
…ra downtime
@BrianCurtis-NOAA
Merge branch 'feature/rt-automation' of https://github.com/BrianCurti…
9fba3a1 
…s-NOAA/ufs-weather-model into feature/rt-automation
@BrianCurtis-NOAA
Merging develop into feature/rt-automation
34ebb42
@BrianCurtis-NOAA
* Logs renamed and saved for a max one month.
aa8cb88 
* Log location sent as comment, not log itself.
* fixed extra code in sh export.
* renamed runFunction to run_function.
* removed extra debug for issue to be addressed later.
@BrianCurtis-NOAA
Merge branch 'feature/rt-automation' of https://github.com/BrianCurti…
44f35f7 
…s-NOAA/ufs-weather-model into feature/rt-automation
@BrianCurtis-NOAA
* typo fixes
c0e52cc 
* exception fixes
* skip-ci
@BrianCurtis-NOAA
Copy link
Collaborator Author

Log Name:rt_auto_hera.intel_20210224184321.log
Log Location:/scratch1/NCEPDEV/nems/Brian.Curtis/git2/BrianCurtis-NOAA/ufs-weather-model-rt/tests/auto
Logs are kept for one month

junwang-noaa
junwang-noaa approved these changes Feb 25, 2021
View reviewed changes
Copy link
Collaborator

@junwang-noaa junwang-noaa left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This PR is the first of the incremental automatic RT commits. It adds two files and does not have any impact to the model.

climbfuji
climbfuji approved these changes Feb 25, 2021
View reviewed changes
Copy link
Collaborator

@climbfuji climbfuji left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am approving based on the fact that it doesn't impact the current code. I am looking forward to the tutorial next week and then I will hopefully be able to give a more educated feedback on the Python script.

tests/auto/rt_auto.sh Show resolved Hide resolved
tests/auto/rt_auto.sh Outdated Show resolved Hide resolved
DeniseWorthen
DeniseWorthen approved these changes Feb 25, 2021
View reviewed changes
Copy link
Collaborator

@DeniseWorthen DeniseWorthen left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approving, on the basis of no impact to the current tests.

@junwang-noaa junwang-noaa merged commit cd755dc into ufs-community:develop Feb 25, 2021
@BrianCurtis-NOAA BrianCurtis-NOAA deleted the feature/rt-automation branch February 26, 2021 16:32
AnningCheng-NOAA added a commit to AnningCheng-NOAA/ufs-weather-model that referenced this pull request Mar 8, 2021
@AnningCheng-NOAA
Merge remote-tracking branch 'upstream/develop' into merra2
e6bc1e4 
* upstream/develop:
  update MOM6 to GFDL 20210224 main branch commit (ufs-community#439)
  Add GNU and Cheyenne Support to Automated RT (ufs-community#444)
  Move Noah MP init to CCPP and update Noah MP regression tests, ice flux init bug fix in CCPP (ufs-community#425)
  Feature/rt automation (ufs-community#403)
  Update ccpp-physics. Make RRTMGP thread safe (ufs-community#418)
  Update regression tests from GFSv15+Thompson to GFSv16+Thompson, include "Add one regional regression test in DEBUG mode. (ufs-community#419)" (ufs-community#421)
  UGWP v0 v1 combined (ufs-community#396)
  add optional mesh in MOM6; add dz_min and min_seaice as configurable variables for coupled model (ufs-community#399)
  updates FMS to 2020.04.01 (ufs-community#392)
  Move LSM vegetation lookup tables into CCPP, clean up RUC snow cover on ice initialization (remove IPD step 2)  (ufs-community#407)
  Update CMEPS for HAFS integration; add datm and coupled-model tests on Gaea (ufs-community#401)
  Remove legacy gnumake build from fv3atm and NEMS, remove legacy Python 2.7 support, rename v16beta to v16 and RT updates (ufs-community#384)
  MOM6 bugfixes, GFDL update, update CDMBGWD settings; fix for restart reproducibility (without waves) when USE_LA_LI2016=True, sign error on fprec passed to ocean, GFDL update, resolution dependent cdmbgwd settings (ufs-community#379)
  dycore options to add zero-gradient BC to reconstruct interface u/v and change dz_min as input (ufs-community#369)
  Update develop from NOAA-GSL: RUC ice, MYNN sfclay, stochastic land perturbations (ufs-community#386)
  update cpl gfsv16 tests, rrtmgp fix and bug fixes in cmeps (ufs-community#378)
  point fv3 to EMC develop branch (ufs-community#377)
  Remove IPD steps 3 and 5 (ufs-community#357)
  Update CMEPS  (ufs-community#345)
  Implementation of CCPP timestep_init and timestep_final phases (ufs-community#337)
  Remove unnecessary SIMD instruction sets for Jet, first round of cleanup in rt.conf, initialize cld_amt to zero for regional runs (dycore) (ufs-community#353)
  add frac grid input, update and add additional cpld tests (ufs-community#354)
  Add checkpoint restarts for ufs-cpld (ufs-community#342)
  Update the format of rt.conf (ufs-community#349)
  Remove IPD (step 1) (ufs-community#331)
  Feature/ww3update (ufs-community#334)
  Replace old regional SDF with FV3_GFS_v15_thompson_mynn (ufs-community#333)
  Update modules with hpc-stack v1.1.0 (ufs-community#319)
  Regression test log for PR ufs-community#323 for jet.intel (ufs-community#336)
  RRTMGP and Thompson MP coupling (ufs-community#323)
  Add 2 new tests for DATM-MOM6-CICE6 application (ufs-community#332)
  Add optional bulk flux calculation in ufs-datm (ufs-community#266)
  Final-final GFS v16 updates / restart reproducibility bugfixes (ufs-community#325)
  Updates to build for JEDI linking/control, add wcoss2 (ufs-community#295)
  Update CICE, Move regression test input outside baseline directory (ufs-community#270)
  Feature/update mom6 and retain b4b results for 025x025 resolution (ufs-community#290)
  Update for Jet, bug fixes in running with frac_grid=T and GFDL MP, and in restarting with frac_grid=T  (ufs-community#304)
  Updates to stochastic_physics_wrapper (ufs-community#280)
  Update develop from gsd/develop 2020/11/20: Unified gravity wave drag, updates to other GSL physics (ufs-community#297)
  Fix to allow quilting with non-factors for layout (ufs-community#250)
  rt update (ufs-community#261)
epic-cicd-jenkins pushed a commit that referenced this pull request Apr 17, 2023
@gsketefian
Bugfixes for default values of grid parameters and forecast computati…
8c34186 
…onal parameters; bugfix for setting FXILAM_NCO_BASEDIR in WE2E tests; other improvements. (#403)
epic-cicd-jenkins pushed a commit that referenced this pull request Apr 17, 2023
Fix quoting/escaping in Jenkinsfile (#403)
045224b 
* Fix quoting/escaping in Jenkinsfile

* Export we2e comprehensive tests var
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@MinsukJi-NOAA MinsukJi-NOAA MinsukJi-NOAA left review comments

@climbfuji climbfuji climbfuji approved these changes

@aerorahul aerorahul aerorahul approved these changes

@junwang-noaa junwang-noaa junwang-noaa approved these changes

@DeniseWorthen DeniseWorthen DeniseWorthen approved these changes

@DusanJovic-NOAA DusanJovic-NOAA DusanJovic-NOAA approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Automate regression testing through triggers in PR comments
7 participants
@BrianCurtis-NOAA @DeniseWorthen @junwang-noaa @climbfuji @aerorahul @DusanJovic-NOAA @MinsukJi-NOAA