WRF detailed setup procedure #689

marcusgaspar · 2022-10-27T16:07:49Z

In this Pull Request I'm detailing all the setup procedures to run and test WRF v4 using Cycle Cloud.
The original setup procedure was not clear enough and there were some missing steps. I took a long time to figure out the missing steps and make it work.
I'm sharing this back to the community as I believe I will be useful for everybody who wants to run a WRF v4 test on Azure using Cycle Cloud.

garvct

Thanks for all these detailed instructions, but azurehpc/apps/* (e.g wrf) should only contain scripts and code to build, install and run applications (independent of cluster deployment). I think the best location for deploying WRF on a cyclecloud cluster would be under the experimental directory.
Would it be possible to update/add the wrf build and install scripts (including creating the wrf data) in azurehpc/wrf and putting the complete deployment of WRF on cyclecloud under the experimental directory?

garvct · 2022-10-31T19:16:51Z

tutorials/cyclecloud/install-cyclecloud.md

@@ -0,0 +1,105 @@
+# Install and Setup CycleCloud for a Lab environment


Does examples/cycleserver_msi and examples/cycleserver deploy VNET and cycleserver automatically via a simple azurehpc config file.
It seems you are deploying the same but with all the manual steps?

Yes, I'm using manual steps. These manual steps can be useful in scenarios:

where people may not want to install it using azurehpc scripts; or

for learning purposes, where people wants to understand what exactly is installed/required.

I can add a mention about examples/cycleserver_msi and examples/cycleserver as an alternative option.

garvct · 2022-10-31T19:19:50Z

apps/wrf/readme.md


+Summary of this procedure:
+- Installs CycleCloud environment from scratch
+- Creates NFS storage server using CycleCloud cluster template


Would Azure netapp files or a PFS be better for production?

Yes, indeed. But, this is a procedure to setup a Lab environment.
I will add comments regarding Lab env and ANF or PFS as options for production.

garvct · 2022-10-31T19:21:58Z

apps/wrf/readme.md

+## Download azurehpc GitHub repository
+cd /data
+#git clone https://github.com/Azure/azurehpc.git
+git clone https://github.com/marcusgaspar/azurehpc.git


Is this URL correct (you are pointing to your fork?)

It is temporary, as I'm currently using my fork during POCs.

garvct · 2022-10-31T19:23:18Z

apps/wrf/readme.md

+mkdir ~/test1
+cd ~/test1
+
+qsub -l select=1:nodearray=execute1:ncpus=60:mpiprocs=60,place=scatter:excl -v "SKU_TYPE=hbv2,INPUTDIR=/apps/hbv2/wrf-openmpi/WRF-4.1.5/run" /data/azurehpc/apps/wrf/run_wrf_openmpi.pbs


For HBv2, should ncpus=120 ?

These were tests I did to measure the execution time with different configs. I forgot to add the execution time duration results. I will add a chart with it.

garvct · 2022-10-31T19:24:29Z

apps/wrf/readme.md

+qsub -l select=1:nodearray=execute1:ncpus=60:mpiprocs=60,place=scatter:excl -v "SKU_TYPE=hbv2,INPUTDIR=/apps/hbv2/wrf-openmpi/WRF-4.1.5/run" /data/azurehpc/apps/wrf/run_wrf_openmpi.pbs
+```
+
+- Test 2 


Why so many tests, is the only difference between each test the number of nodes (select=N)?

These were tests I did to measure the execution time with different configs. I forgot to add the execution time duration results. I will add a chart with it.

garvct · 2022-10-31T19:27:09Z

apps/wrf/readme.md

+mkdir ~/test5
+cd ~/test5
+
+qsub -l select=3:nodearray=execute1:ncpus=60:mpiprocs=60,place=scatter:excl -v "SKU_TYPE=hbv2,INPUTDIR=/apps/hbv2/wrf-openmpi/WRF-4.1.5/run" /data/azurehpc/apps/wrf/run_wrf_openmpi.pbs


For HB120rs_v3, ncpus=120 ?, there is also references to hbv2 ?

During my tests, I've used hbv2 reference and I was able to perform tests successfully on HBv2 and HBv3.
Do you recommend changing to hbv3 reference when running on HBv3?
If I change it, do I need to run the WRF and WPS build again?

yes, it would be better to use HBv3 (but not absolutely necessary, latest is now HBv4, it will keep changing)

garvct · 2022-10-31T19:28:13Z

apps/wrf/readme.md

+mkdir ~/test6
+cd ~/test6
+
+qsub -l select=3:nodearray=execute1:ncpus=64:mpiprocs=64,place=scatter:excl -v "SKU_TYPE=hbv2,INPUTDIR=/apps/hbv2/wrf-openmpi/WRF-4.1.5/run" /data/azurehpc/apps/wrf/run_wrf_openmpi.pbs


HB120-64rs_v3 test, but hbv2 references?

Same above. During my tests, I've used hbv2 reference and I was able to perform tests successfully on HBv2 and HBv3.
Do you recommend changing to hbv3 reference when running on HBv3?
If I change it, do I need to run the WRF and WPS build again?

I think it's good practice to build on the SKU you are running on. It's confusing to run on HBv3 but reference hbv2. To simply to documentation, I would just pick HBv3 (because its newer than hbv2) and give a few examples running specifically on HBv3. You could then add a note to state that a very similar procedure can be also used to run WRF on hbv2.

garvct · 2022-12-05T20:48:02Z

apps/wrf/readme.md

 ## Prerequisites

-Cluster is built with the desired configuration for networking, storage, compute etc. You can see the tutorial or examples folder in this repo for how to set this up. Spack is installed (See [here](../spack/readme.md) for details).
+- You need a cluster built with the desired configuration for networking, storage, compute etc. You can use the tutorial in this repo with an end-to-end instructions to setup a lab environment on Cycle Cloud to run WPS and WRF v4. (See [Install and run WPS and WRF v4 - end-to-end setup guide](../../experimental/wrf_on_cyclecloud/readme.md) for details). 


Please link to master branch and not your branch.

garvct · 2022-12-05T20:49:20Z

apps/wrf/readme.md

 ## Prerequisites

-Cluster is built with the desired configuration for networking, storage, compute etc. You can see the tutorial or examples folder in this repo for how to set this up. Spack is installed (See [here](../spack/readme.md) for details).
+- You need a cluster built with the desired configuration for networking, storage, compute etc. You can use the tutorial in this repo with an end-to-end instructions to setup a lab environment on Cycle Cloud to run WPS and WRF v4. (See [Install and run WPS and WRF v4 - end-to-end setup guide](../../experimental/wrf_on_cyclecloud/readme.md) for details). 
+- As this procedure uses HBv2 VMs to run WRFv4 simulations, you may need to request quota increase for this type of SKU in the subscription and region you will deploy the environment. You can use different SKU if you want to.


May be worth mentioning that other specialty VM's can also work with some minor modifications.

garvct · 2022-12-05T20:56:54Z

apps/wrf/readme.md

+mkdir ~/test1
+cd ~/test1
+
+qsub -l select=1:nodearray=execute1:ncpus=60:mpiprocs=30,place=scatter:excl -v "SKU_TYPE=hbv2,INPUTDIR=/apps/hbv2/wrf-openmpi/WRF-4.1.5/run" /data/azurehpc/apps/wrf/run_wrf_openmpi.pbs


For HBv2, with the new NUMA topology (30 Numa --> 4 numa), the only ncpus or mpiprocs that would make sense would be 32, 64, 96 or 120). I do not think 60 and 30 is optimal with new Numa topology?

garvct · 2022-12-05T20:58:33Z

apps/wrf/readme.md

+mkdir ~/test2
+cd ~/test2
+
+qsub -l select=2:nodearray=execute1:ncpus=60:mpiprocs=30,place=scatter:excl -v "SKU_TYPE=hbv2,INPUTDIR=/apps/hbv2/wrf-openmpi/WRF-4.1.5/run" /data/azurehpc/apps/wrf/run_wrf_openmpi.pbs


See comment above about 30 and 60 ncpus/mpiprocs on HBv2 with 3 NUMA domains.

garvct · 2022-12-05T20:58:46Z

apps/wrf/readme.md

+mkdir ~/test3
+cd ~/test3
+
+qsub -l select=3:nodearray=execute1:ncpus=60:mpiprocs=30,place=scatter:excl -v "SKU_TYPE=hbv2,INPUTDIR=/apps/hbv2/wrf-openmpi/WRF-4.1.5/run" /data/azurehpc/apps/wrf/run_wrf_openmpi.pbs


See same comment about.

garvct · 2022-12-05T21:03:32Z

apps/wrf/run_real_openmpi.pbs

+
+#### Run real.exe
+echo "-- Run real.exe"
+mpirun $mpi_options -n $NPROCS --hostfile $PBS_NODEFILE --bind-to numa ./real.exe


Now, with HBv2 new numa topology --bind-to l3cache may be more optimal?

garvct · 2022-12-05T21:03:54Z

apps/wrf/run_wps-real_openmpi.pbs

+
+#### Run real.exe
+echo "-- Run real.exe"
+mpirun $mpi_options -n $NPROCS --hostfile $PBS_NODEFILE --bind-to numa ./real.exe


Same comment as above.

garvct · 2022-12-05T21:09:23Z

experimental/wrf_on_cyclecloud/readme.md

+```
+
+Follow the procedures [here](https://docs.microsoft.com/en-us/azure/cyclecloud/tutorials/modify-cluster-template?view=cyclecloud-8#import-the-new-cluster-template) to upload the Cycle Cloud custom template created for WRF.
+Use the template: [opbswrf-template.txt](opbswrf-template.txt) 


Please change the link to mater instead of fork.

garvct · 2022-12-05T21:10:07Z

experimental/wrf_on_cyclecloud/readme.md

+### Install WPS/WRF v4 software (via “azurehpc” scripts)
+Now you have a WRF cluster properly configured and running in Cycle Cloud. 
+
+You can follow the instructions here to finish the WPS/WRF installation: [Install and run WPS and WRF v4 - Setup guide](/apps/wrf/readme.md) 


Please change link to master branch instead of forked branch

garvct · 2022-12-05T21:14:24Z

apps/wrf/readme.md

+
+### Test Results
+In the graph below you can compare the execution time from tests performed, with different number of nodes, cores, mpicores and SKUs:
+![Import-Template1](images/wrf-test-results.png)


In chart ncpus/mpiprocs=60, do not make sense for HBv2 with 4 Numa domains.? (See comments above)

fix permissions related to managed identity on the subscription

marcusgaspar added 27 commits October 20, 2022 19:24

Update readme.md

24d404e

Update readme.md

fd66f10

upload images

df79228

upload images

8516c5b

Upload setup guide

6f2543f

Update pre-reqs

6dd1217

Install cyclecloud setup guide

23357a5

update

fcbf885

fix

df856db

initialize CycleCloud CLI

b907867

install cyclecloud

36882ed

remove pre-reqs

b2062d5

update pre-reqs

4fbf835

formatt

ff16a4f

updates and download_gfs_files.py

10f70f9

updates

6851aad

change repo

754efe0

updates

db35ea8

updates

dae9123

update

88f3c75

changes

71996ae

changes

63f63cd

fix test folder

d335434

change permission

8c652ac

validation

c5d114e

test scenarios

ffa3024

changes

ea81743

xpillons requested a review from garvct October 27, 2022 16:28

granting write permission to group on /data /apps

34178e3

garvct requested changes Oct 31, 2022

View reviewed changes

marcusgaspar added 6 commits December 2, 2022 14:46

adjustments required for pull request

988a1f2

small changes

52e451e

fix link

a480bd4

fix link

74e1706

link

d9227ad

fix repo

b65cbed

garvct requested changes Dec 5, 2022

View reviewed changes

marcusgaspar added 6 commits December 6, 2022 08:49

clean up

642b875

temporary change while donte complete the PR.

79516f5

Update install-cyclecloud.md

2ea3d5b

fix permissions related to managed identity on the subscription

Update install-cyclecloud.md

e1f0428

Update install-cyclecloud.md

8c0604d

Merge branch 'Azure:master' into master

5b6737a

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

WRF detailed setup procedure #689

WRF detailed setup procedure #689

marcusgaspar commented Oct 27, 2022

garvct left a comment •

edited

garvct Oct 31, 2022

marcusgaspar Nov 1, 2022

garvct Oct 31, 2022 •

edited

marcusgaspar Nov 1, 2022 •

edited by garvct

garvct Oct 31, 2022

marcusgaspar Nov 1, 2022

garvct Oct 31, 2022

marcusgaspar Nov 1, 2022

garvct Oct 31, 2022

marcusgaspar Nov 1, 2022

garvct Oct 31, 2022

marcusgaspar Nov 1, 2022

garvct Dec 5, 2022

garvct Oct 31, 2022

marcusgaspar Nov 1, 2022

garvct Nov 2, 2022

garvct Dec 5, 2022

garvct Dec 5, 2022

garvct Dec 5, 2022

garvct Dec 5, 2022

garvct Dec 5, 2022

garvct Dec 5, 2022

garvct Dec 5, 2022

garvct Dec 5, 2022

garvct Dec 5, 2022

garvct Dec 5, 2022

		@@ -0,0 +1,105 @@
		# Install and Setup CycleCloud for a Lab environment

WRF detailed setup procedure #689

Are you sure you want to change the base?

WRF detailed setup procedure #689

Conversation

marcusgaspar commented Oct 27, 2022

garvct left a comment • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

garvct Oct 31, 2022 • edited

Choose a reason for hiding this comment

marcusgaspar Nov 1, 2022 • edited by garvct

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

garvct left a comment •

edited

garvct Oct 31, 2022 •

edited

marcusgaspar Nov 1, 2022 •

edited by garvct