From 5b3c9e2225458e5741ed1ce34dc413b3ce7276e4 Mon Sep 17 00:00:00 2001 From: bnb32 Date: Sat, 16 Mar 2024 08:42:21 -0600 Subject: [PATCH 1/7] sup3rwind examples --- examples/sup3rwind/README.rst | 60 +++++ examples/sup3rwind/env.yml | 228 ++++++++++++++++++ .../run_configs/trhp/config_collect_h5.json | 27 +++ .../run_configs/trhp/config_fwp_spatial.json | 40 +++ .../run_configs/trhp/config_pipeline.json | 15 ++ .../run_configs/wind/config_collect_h5.json | 32 +++ .../run_configs/wind/config_collect_nc.json | 31 +++ .../run_configs/wind/config_fwp_spatial.json | 53 ++++ .../run_configs/wind/config_fwp_temporal.json | 37 +++ .../run_configs/wind/config_pipeline.json | 23 ++ sup3r/postprocessing/file_handling.py | 49 +++- 11 files changed, 582 insertions(+), 13 deletions(-) create mode 100644 examples/sup3rwind/README.rst create mode 100644 examples/sup3rwind/env.yml create mode 100644 examples/sup3rwind/run_configs/trhp/config_collect_h5.json create mode 100644 examples/sup3rwind/run_configs/trhp/config_fwp_spatial.json create mode 100755 examples/sup3rwind/run_configs/trhp/config_pipeline.json create mode 100644 examples/sup3rwind/run_configs/wind/config_collect_h5.json create mode 100644 examples/sup3rwind/run_configs/wind/config_collect_nc.json create mode 100644 examples/sup3rwind/run_configs/wind/config_fwp_spatial.json create mode 100644 examples/sup3rwind/run_configs/wind/config_fwp_temporal.json create mode 100755 examples/sup3rwind/run_configs/wind/config_pipeline.json diff --git a/examples/sup3rwind/README.rst b/examples/sup3rwind/README.rst new file mode 100644 index 000000000..bc285db0e --- /dev/null +++ b/examples/sup3rwind/README.rst @@ -0,0 +1,60 @@ +################ +Sup3rWind Examples +################ + +Super-Resolution for Renewable Energy Resource Data with Wind from Reanalysis Data (Sup3rWind) is one application of the sup3r software. In this work, we train generative models to create high-resolution (2km 5-minute) wind data based on coarse (30km hourly) ERA5 data. The high-resolution output data is publicly available via the `Open Energy Data Initiative (OEDI) `_ and via HSDS at the bucket ``nrel-pds-hsds`` and path ``/nrel/wtk/sup3rwind``. This data covers recent historical time periods for an expanding selection of countries. + +Sup3rWind Data Access +=================== + +The Sup3rWind data is publicly available in a public AWS S3 bucket. The data files can be downloaded directly from there to your local machine or an EC2 instance using the `OEDI data explorer `_ or the `AWS CLI `_. A word of caution: there's a lot of data here. The smallest Sup3rWind file for just a single variable at 2-km 5-minute resolution is 130 GB. + +The Sup3rWind data is also loaded into `HSDS `_ so that you may stream the data via the `NREL developer API `_ or your own HSDS server. This is the best option if you're not going to want a full annual dataset. See these `rex instructions `_ for more details on how to access this data with HSDS and rex. + +Example Sup3rWind Data Usage +========================== + +Sup3rWind data can be used in generally the same way as Sup3rCC data, with the condition that Sup3rWind includes only wind data and ancillary variables for modeling wind energy generation. Refer to ``/sup3r/examples/sup3rcc/README.rst`` for usage patterns. + +Running Sup3rWind Models +====================== + +The process for running the Sup3rWind models is much the same as for Sup3rCC (``sup3r/examples/sup3rcc/README.rst``). + +#. Download the Sup3rWind models to your hardware using the AWS CLI: ``$ aws s3 cp s3://nrel-pds-wtk/sup3rwind/models/`` +#. Download the ERA5 data that you want to downscale from `ERA5-single-levels `_ and/or `ERA5-pressure-levels `_. +#. Setup the Sup3rWind software. We recommend using `miniconda `_ to manage your python environments. You can create a sup3r environment with the conda file in this example directory: ``$ conda env create -n sup3rwind --file env.yml`` +#. Copy this examples directory to your hardware. You're going to be using the folder structure in ``/sup3r/examples/sup3rwind/run_configs`` as your project directories (``/sup3r/`` is a git clone of the sup3r software repo). +#. Navigate to ``/sup3r/examples/sup3rwind/run_configs/wind/`` and/or ``sup3r/examples/sup3rwind/run_configs/trhp`` and update all of the filepaths in the config files for the source ERA5 data, Sup3rWind models, and exogenous data sources (e.g. the ``topography`` source file). +#. Update the execution control parameters in the ``config_fwp_spatial.json`` file based on the hardware you're running on. +#. Run ``sup3r-pipeline`` to run just one job. There are also batch options for running multiple jobs, but we recommend starting with ``sup3r-pipeline`` (more on the sup3r CLIs `here `_). +#. To run ``sup3r-pipeline``, make sure you are in the directory with the ``config_pipeline.json`` and ``config_fwp_spatial.json`` files, and then run this command: ``python -m sup3r.cli -c config_pipeline.json pipeline`` +#. If you're running on a slurm cluster, this will kick off a number of jobs that you can see with the ``squeue`` command. If you're running locally, your terminal should now be running the Sup3rWind models. The software will create a ``./logs/`` directory in which you can monitor the progress of your jobs. +#. The ``sup3r-pipeline`` is designed to run several modules in serial, with each module running multiple chunks in parallel. Once the first module (forward-pass) finishes, you'll want to run ``python -m sup3r.cli -c config_pipeline.json pipeline`` again. This will clean up status files and kick off the next step in the pipeline (if the current step was successful). + +Sup3rWind Versions +================ + +The Sup3rWind data has versions that coincide with the sup3r software versions. Note that not every sup3r software version will have a corresponding Sup3rWind data release, but every Sup3rWind data release will have a corresponding sup3r software version. + +.. list-table:: + :widths: auto + :header-rows: 1 + + * - Version + - Effective Date + - Notes + * - 0.1.2 + - 3/15/2024 + - Initial release of Sup3rWind for Ukraine, Moldova, and part of Romania. Includes 2-km 5-minute wind speed and wind direction data and 2-km hourly wind speed, wind direction, pressure, temperature, and relative humidity data for 2000-2023. + + +Recommended Citation +==================== + +Brandon Benton, Grant Buster, Pavlo Pinchuk, Andrew Glaws, Ryan King, Galen Maclaurin, Ilya Chernyakhovskiy. "Super-Resolution for Renewable Energy Resource Data with Wind from Reanalysis Data (Sup3rWind)". + +Acknowledgements +================ + +This work was authored by the National Renewable Energy Laboratory, operated by Alliance for Sustainable Energy, LLC, for the U.S. Department of Energy (DOE) under Contract No. DE-AC36-08GO28308. Funding provided by the DOE Grid Deployment Office (GDO), the DOE Advanced Scientific Computing Research (ASCR) program, the DOE Solar Energy Technologies Office (SETO), and the Laboratory Directed Research and Development (LDRD) program at the National Renewable Energy Laboratory. The research was performed using computational resources sponsored by the DOE Office of Energy Efficiency and Renewable Energy and located at the National Renewable Energy Laboratory. The views expressed in the article do not necessarily represent the views of the DOE or the U.S. Government. The U.S. Government retains and the publisher, by accepting the article for publication, acknowledges that the U.S. Government retains a nonexclusive, paid-up, irrevocable, worldwide license to publish or reproduce the published form of this work, or allow others to do so, for U.S. Government purposes. diff --git a/examples/sup3rwind/env.yml b/examples/sup3rwind/env.yml new file mode 100644 index 000000000..526a7fba0 --- /dev/null +++ b/examples/sup3rwind/env.yml @@ -0,0 +1,228 @@ +absl-py==1.4.0 +addfips==0.4.0 +affine==2.4.0 +alabaster==0.7.13 +alphashape==1.3.1 +anyio==4.0.0 +argon2-cffi==23.1.0 +argon2-cffi-bindings==21.2.0 +arrow==1.3.0 +asttokens @ file:///home/conda/feedstock_root/build_artifacts/asttokens_1694046349000/work +astunparse==1.6.3 +async-lru==2.0.4 +attrs==23.1.0 +Babel==2.12.1 +backcall @ file:///home/conda/feedstock_root/build_artifacts/backcall_1592338393461/work +backports.functools-lru-cache @ file:///home/conda/feedstock_root/build_artifacts/backports.functools_lru_cache_1687772187254/work +beautifulsoup4==4.12.2 +bleach==6.1.0 +cachetools==5.3.1 +Cartopy @ file:///croot/cartopy_1674677761573/work +cdsapi==0.6.1 +certifi @ file:///home/conda/feedstock_root/build_artifacts/certifi_1700303426725/work/certifi +cffi==1.16.0 +cftime==1.6.2 +charset-normalizer==3.2.0 +click==8.1.7 +click-log==0.4.0 +click-plugins==1.1.1 +cligj==0.7.2 +cloudpickle==2.2.1 +colorama==0.4.6 +comm @ file:///home/conda/feedstock_root/build_artifacts/comm_1691044910542/work +contourpy @ file:///opt/conda/conda-bld/contourpy_1663827406301/work +cycler @ file:///tmp/build/80754af9/cycler_1637851556182/work +dask==2023.5.0 +debugpy @ file:///croot/debugpy_1690905042057/work +decorator @ file:///home/conda/feedstock_root/build_artifacts/decorator_1641555617451/work +defusedxml==0.7.1 +docutils==0.18.1 +entrypoints @ file:///home/conda/feedstock_root/build_artifacts/entrypoints_1643888246732/work +exceptiongroup==1.1.3 +executing @ file:///home/conda/feedstock_root/build_artifacts/executing_1667317341051/work +fastjsonschema==2.18.1 +fiona==1.9.5 +flatbuffers==23.5.26 +fonttools==4.25.0 +fqdn==1.5.1 +fsspec==2023.9.0 +gast==0.4.0 +geopandas==0.13.2 +google-auth==2.23.0 +google-auth-oauthlib==0.4.6 +google-pasta==0.2.0 +grpcio==1.58.0 +h5py==3.9.0 +h5pyd==0.15.1 +idna==3.4 +imageio==2.31.5 +imagesize==1.4.1 +importlib-metadata==6.8.0 +importlib-resources==6.0.1 +iniconfig==2.0.0 +ipykernel @ file:///home/conda/feedstock_root/build_artifacts/ipykernel_1693880262622/work +ipython @ file:///home/conda/feedstock_root/build_artifacts/ipython_1680185408135/work +ipywidgets==8.1.1 +isoduration==20.11.0 +jedi @ file:///home/conda/feedstock_root/build_artifacts/jedi_1696326070614/work +Jinja2==3.1.2 +joblib==1.3.2 +json5==0.9.14 +jsonpointer==2.4 +jsonschema==4.19.2 +jsonschema-specifications==2023.7.1 +jupyter-events==0.9.0 +jupyter-lsp==2.2.0 +jupyter_client==8.6.0 +jupyter_core @ file:///home/conda/feedstock_root/build_artifacts/jupyter_core_1696972407098/work +jupyter_server==2.10.0 +jupyter_server_terminals==0.4.4 +jupyterlab==4.0.8 +jupyterlab-pygments==0.2.2 +jupyterlab-widgets==3.0.9 +jupyterlab_server==2.25.0 +kaleido==0.2.1 +keras==2.8.0 +Keras-Preprocessing==1.1.2 +kiwisolver @ file:///croot/kiwisolver_1672387140495/work +lazy_loader==0.3 +libclang==16.0.6 +locket==1.0.0 +Markdown==3.4.4 +MarkupSafe==2.1.3 +matplotlib @ file:///croot/matplotlib-suite_1693812469450/work +matplotlib-inline @ file:///home/conda/feedstock_root/build_artifacts/matplotlib-inline_1660814786464/work +-e git+ssh://git@github.com/bnb32/misc_tools.git@016334c0fe767158726913ba8a8db2f6811be0b4#egg=misc_tools +mistune==3.0.2 +mkl-fft @ file:///croot/mkl_fft_1695058164594/work +mkl-random @ file:///croot/mkl_random_1695059800811/work +mkl-service==2.4.0 +munkres==1.1.4 +nbclient==0.8.0 +nbconvert==7.11.0 +nbformat==5.9.2 +nest-asyncio @ file:///home/conda/feedstock_root/build_artifacts/nest-asyncio_1697083700168/work +netCDF4==1.5.8 +networkx==3.1 +notebook==7.0.6 +notebook_shim==0.2.3 +NREL-farms==1.0.5 +NREL-gaps==0.6.1 +NREL-NRWAL==0.0.11 +-e git+ssh://git@github.com/NREL/phygnn.git@da93661077c42f36a2d613a63e1f499994cbc4b0#egg=NREL_phygnn +NREL-PySAM==4.1.0 +NREL-reV==0.7.3 +NREL-reVX==0.3.53 +-e git+ssh://git@github.com/NREL/rex.git@7d724b10222c22a81fb9df8834ae8575a28f6ead#egg=NREL_rex +-e git+ssh://git@github.com/NREL/sup3r.git@8a02005db703e456c5a7f1879203a3ffb16c3fe9#egg=NREL_sup3r +numpy==1.22.0 +numpydoc==1.5.0 +oauthlib==3.2.2 +opt-einsum==3.3.0 +overrides==7.4.0 +packaging==23.1 +pandas==2.0.3 +pandocfilters==1.5.0 +parso @ file:///home/conda/feedstock_root/build_artifacts/parso_1638334955874/work +partd==1.4.0 +pexpect @ file:///home/conda/feedstock_root/build_artifacts/pexpect_1667297516076/work +pickleshare @ file:///home/conda/feedstock_root/build_artifacts/pickleshare_1602536217715/work +Pillow==10.0.0 +pkgutil_resolve_name==1.3.10 +platformdirs @ file:///home/conda/feedstock_root/build_artifacts/platformdirs_1696272223550/work +plotly==5.16.1 +plotting==0.0.7 +pluggy==1.3.0 +prometheus-client==0.18.0 +prompt-toolkit @ file:///home/conda/feedstock_root/build_artifacts/prompt-toolkit_1688565951714/work +protobuf==3.20.3 +psutil==5.9.5 +psycopg2-binary==2.9.9 +ptyprocess @ file:///home/conda/feedstock_root/build_artifacts/ptyprocess_1609419310487/work/dist/ptyprocess-0.7.0-py2.py3-none-any.whl +pure-eval @ file:///home/conda/feedstock_root/build_artifacts/pure_eval_1642875951954/work +pyasn1==0.5.0 +pyasn1-modules==0.3.0 +pycparser==2.21 +Pygments @ file:///home/conda/feedstock_root/build_artifacts/pygments_1691408637400/work +pyjson5==1.6.4 +PyJWT==2.8.0 +pyparsing @ file:///opt/conda/conda-bld/pyparsing_1661452539315/work +pyproj @ file:///croot/pyproj_1675244161691/work +pyshp @ file:///tmp/build/80754af9/pyshp_1610641829514/work +pytest==7.4.2 +python-dateutil @ file:///home/conda/feedstock_root/build_artifacts/python-dateutil_1626286286081/work +python-dotenv==1.0.0 +python-json-logger==2.0.7 +pytz==2023.3.post1 +PyWavelets==1.4.1 +PyYAML==6.0.1 +pyzmq @ file:///croot/pyzmq_1686601365461/work +rasterio==1.3.9 +referencing==0.30.2 +requests==2.31.0 +requests-oauthlib==1.3.1 +requests-unixsocket==0.3.0 +rfc3339-validator==0.1.4 +rfc3986-validator==0.1.1 +rioxarray==0.13.4 +rpds-py==0.12.0 +rsa==4.9 +Rtree==1.1.0 +ruff==0.1.3 +scikit-image==0.21.0 +scikit-learn==1.3.0 +scipy==1.10.1 +seaborn==0.12.2 +Send2Trash==1.8.2 +Shapely==1.8.5.post1 +six @ file:///home/conda/feedstock_root/build_artifacts/six_1620240208055/work +sniffio==1.3.0 +snowballstemmer==2.2.0 +snuggs==1.4.7 +soupsieve==2.5 +Sphinx==7.1.2 +sphinx-click==5.0.1 +sphinx-copybutton==0.5.2 +sphinx-tabs==3.4.1 +sphinxcontrib-applehelp==1.0.4 +sphinxcontrib-devhelp==1.0.2 +sphinxcontrib-htmlhelp==2.0.1 +sphinxcontrib-jsmath==1.0.1 +sphinxcontrib-qthelp==1.0.3 +sphinxcontrib-serializinghtml==1.1.5 +stack-data @ file:///home/conda/feedstock_root/build_artifacts/stack_data_1669632077133/work +tabulate==0.9.0 +tenacity==8.2.3 +tensorboard==2.8.0 +tensorboard-data-server==0.6.1 +tensorboard-plugin-wit==1.8.1 +tensorflow==2.8.0 +tensorflow-estimator==2.13.0 +tensorflow-io-gcs-filesystem==0.34.0 +termcolor==2.3.0 +terminado==0.17.1 +tf-estimator-nightly==2.8.0.dev2021122109 +threadpoolctl==3.2.0 +tifffile==2023.7.10 +tinycss2==1.2.1 +toml==0.10.2 +tomli==2.0.1 +toolz==0.12.0 +tornado==6.3.3 +tqdm==4.66.1 +traitlets @ file:///home/conda/feedstock_root/build_artifacts/traitlets_1696377679271/work +trimesh==4.0.2 +types-python-dateutil==2.8.19.14 +typing_extensions==4.5.0 +tzdata==2023.3 +uri-template==1.3.0 +urllib3==1.26.16 +wcwidth @ file:///home/conda/feedstock_root/build_artifacts/wcwidth_1696255154857/work +webcolors==1.13 +webencodings==0.5.1 +websocket-client==1.6.4 +Werkzeug==2.3.7 +widgetsnbextension==4.0.9 +wrapt==1.15.0 +xarray==2023.1.0 +zipp @ file:///croot/zipp_1672387121353/work diff --git a/examples/sup3rwind/run_configs/trhp/config_collect_h5.json b/examples/sup3rwind/run_configs/trhp/config_collect_h5.json new file mode 100644 index 000000000..1604dc0e3 --- /dev/null +++ b/examples/sup3rwind/run_configs/trhp/config_collect_h5.json @@ -0,0 +1,27 @@ +{ + "n_writes": 3, + "dset_split": false, + "join_times": true, + "execution_control": { + "memory": 175, + "option": "kestrel", + "walltime": 40, + "feature": "--qos=normal", + "alloc": "usaiddata" + }, + "features": [ + "temperature_2m", + "temperature_100m", + "temperature_200m", + "relativehumidity_2m", + "relativehumidity_100m", + "relativehumidity_200m", + "pressure_0m", + "pressure_100m", + "pressure_200m" + ], + "file_paths": "./output/fwp_out_*.h5", + "log_file": "./collect_h5.log", + "log_level": "DEBUG", + "out_file": "./output/sup3rwind_ancillary.h5" +} diff --git a/examples/sup3rwind/run_configs/trhp/config_fwp_spatial.json b/examples/sup3rwind/run_configs/trhp/config_fwp_spatial.json new file mode 100644 index 000000000..59dd1fd76 --- /dev/null +++ b/examples/sup3rwind/run_configs/trhp/config_fwp_spatial.json @@ -0,0 +1,40 @@ +{ + "worker_kwargs": {"pass_workers": 5, "ti_workers": 1}, + "input_handler_kwargs": { + "worker_kwargs": {"ti_workers": 1}, + "res_kwargs": {"concat_dim": "time", + "combine": "nested"}}, + "input_handler": "DataHandlerNC", + "max_nodes": 30, + "execution_control": { + "memory": 174, + "option": "kestrel", + "alloc": "usaiddata", + "walltime": 47 + }, + "file_paths": [ + "PLACEHOLDER" + ], + "fwp_chunk_shape": [ + 75, + 75, + 38 + ], + "model_class": "SurfaceSpatialMetModel", + "log_level": "DEBUG", + "log_pattern": "./logs/fwp_spatial_log_{node_index}.log", + "model_kwargs": "./sup3rwind_models_202401/sup3rwind_trhp_step1_15x_1x_9f", + "out_pattern": "./output/fwp_out_{file_id}.h5", + "spatial_pad": 0, + "temporal_pad": 5, + "exo_kwargs": { + "topography": { + "file_paths": [ + "PLACEHOLDER" + ], + "source_file": "PLACEHOLDER", + "exo_resolution": {"spatial": "1km", "temporal": "60min"}, + "steps": [{"model": 0, "combine_type": "input"}, + {"model": 0, "combine_type": "output"}]} + } +} diff --git a/examples/sup3rwind/run_configs/trhp/config_pipeline.json b/examples/sup3rwind/run_configs/trhp/config_pipeline.json new file mode 100755 index 000000000..06ef3dbb1 --- /dev/null +++ b/examples/sup3rwind/run_configs/trhp/config_pipeline.json @@ -0,0 +1,15 @@ +{ + "logging": { + "log_level": "DEBUG" + }, + "pipeline": [ + { + "forward-pass-spatial": "./config_fwp_spatial.json", + "command": "forward-pass" + }, + { + "data-collect-h5": "./config_collect_h5.json", + "command": "data-collect" + } + ] +} diff --git a/examples/sup3rwind/run_configs/wind/config_collect_h5.json b/examples/sup3rwind/run_configs/wind/config_collect_h5.json new file mode 100644 index 000000000..ea18ee1d1 --- /dev/null +++ b/examples/sup3rwind/run_configs/wind/config_collect_h5.json @@ -0,0 +1,32 @@ +{ + "n_writes": 3, + "dset_split": true, + "join_times": true, + "execution_control": { + "memory": 175, + "option": "kestrel", + "walltime": 40, + "feature": "--qos=normal", + "alloc": "usaiddata" + }, + "features": [ + "windspeed_10m", + "windspeed_40m", + "windspeed_80m", + "windspeed_100m", + "windspeed_120m", + "windspeed_160m", + "windspeed_200m", + "winddirection_10m", + "winddirection_40m", + "winddirection_80m", + "winddirection_100m", + "winddirection_120m", + "winddirection_160m", + "winddirection_200m" + ], + "file_paths": "./temporal/output/fwp_out_2000*.h5", + "log_file": "./logs/collect_h5.log", + "log_level": "DEBUG", + "out_file": "./output/sup3rwind_2000.h5" +} diff --git a/examples/sup3rwind/run_configs/wind/config_collect_nc.json b/examples/sup3rwind/run_configs/wind/config_collect_nc.json new file mode 100644 index 000000000..f0557f9fb --- /dev/null +++ b/examples/sup3rwind/run_configs/wind/config_collect_nc.json @@ -0,0 +1,31 @@ +{ + "res_kwargs": {"concat_dim": "Time", "combine": "nested"}, + "execution_control": { + "memory": 175, + "option": "kestrel", + "walltime": 40, + "feature": "--qos=normal", + "alloc": "usaiddata" + }, + "overwrite": false, + "features": [ + "u_10m", + "u_40m", + "u_80m", + "u_100m", + "u_120m", + "u_160m", + "u_200m", + "v_10m", + "v_40m", + "v_80m", + "v_100m", + "v_120m", + "v_160m", + "v_200m" + ], + "file_paths": "./spatial/output/fwp_out_2000*.nc", + "log_file": "./logs/collect_nc.log", + "log_level": "DEBUG", + "out_file": "./output/sup3rwind_2000_hourly.nc" +} diff --git a/examples/sup3rwind/run_configs/wind/config_fwp_spatial.json b/examples/sup3rwind/run_configs/wind/config_fwp_spatial.json new file mode 100644 index 000000000..469c0c16c --- /dev/null +++ b/examples/sup3rwind/run_configs/wind/config_fwp_spatial.json @@ -0,0 +1,53 @@ +{ + "worker_kwargs": {"pass_workers": 5, "ti_workers": 1}, + "input_handler_kwargs": { + "worker_kwargs": {"ti_workers": 1}, + "res_kwargs": {"concat_dim": "time", + "combine": "nested"}}, + "input_handler": "DataHandlerNC", + "bias_correct_method": "monthly_local_linear_bc", + "bias_correct_kwargs": { + "U_10m": {"smoothing": 0, "feature_name": "U_10m", "bias_fp": "/kfs2/shared-projects/sup3r/bc_data/ukraine/bc_files/u_10m_new.h5", "out_range": [-100, 100]}, + "V_10m": {"smoothing": 0, "feature_name": "V_10m", "bias_fp": "/kfs2/shared-projects/sup3r/bc_data/ukraine/bc_files/v_10m_new.h5", "out_range": [-100, 100]}, + "U_100m": {"smoothing": 0, "feature_name": "U_100m", "bias_fp": "/kfs2/shared-projects/sup3r/bc_data/ukraine/bc_files/u_100m_new.h5", "out_range": [-100, 100]}, + "V_100m": {"smoothing": 0, "feature_name": "V_100m", "bias_fp": "/kfs2/shared-projects/sup3r/bc_data/ukraine/bc_files/v_100m_new.h5", "out_range": [-100, 100]}, + "U_200m": {"smoothing": 0, "feature_name": "U_200m", "bias_fp": "/kfs2/shared-projects/sup3r/bc_data/ukraine/bc_files/u_200m_new.h5", "out_range": [-100, 100]}, + "V_200m": {"smoothing": 0, "feature_name": "V_200m", "bias_fp": "/kfs2/shared-projects/sup3r/bc_data/ukraine/bc_files/v_200m_new.h5", "out_range": [-100, 100]} + }, + "max_nodes": 30, + "execution_control": { + "memory": 174, + "option": "kestrel", + "alloc": "usaiddata", + "walltime": 47 + }, + "file_paths": [ + "PLACEHOLDER" + ], + "fwp_chunk_shape": [ + 75, + 75, + 38 + ], + "model_class": "MultiStepGan", + "log_level": "DEBUG", + "log_pattern": "./logs/fwp_spatial_log_{node_index}.log", + "model_kwargs": {"model_dirs": [ + "./sup3rwind_models_202401/sup3rwind_wind_step1_3x_1x_14f", + "./sup3rwind_models_202401/sup3rwind_wind_step2_5x_1x_14f"]}, + "out_pattern": "./spatial/output/fwp_out_{file_id}.nc", + "spatial_pad": 0, + "temporal_pad": 5, + "exo_kwargs": { + "topography": { + "file_paths": [ + "PLACEHOLDER" + ], + "source_file": "PLACEHOLDER", + "exo_resolution": {"spatial": "1km", "temporal": "60min"}, + "steps": [{"model": 0, "combine_type": "input"}, + {"model": 0, "combine_type": "layer"}, + {"model": 1, "combine_type": "input"}, + {"model": 1, "combine_type": "layer"}]} + } +} diff --git a/examples/sup3rwind/run_configs/wind/config_fwp_temporal.json b/examples/sup3rwind/run_configs/wind/config_fwp_temporal.json new file mode 100644 index 000000000..387a7462c --- /dev/null +++ b/examples/sup3rwind/run_configs/wind/config_fwp_temporal.json @@ -0,0 +1,37 @@ +{ + "worker_kwargs": {"pass_workers": 6, "ti_workers": 1}, + "input_handler_kwargs": { + "worker_kwargs": {"ti_workers": 1}, + "res_kwargs": {"concat_dim": "Time", + "combine": "nested"}}, + "input_handler": "DataHandlerNC", + "max_nodes": 30, + "execution_control": { + "memory": 174, + "option": "kestrel", + "alloc": "usaiddata", + "walltime": 47 + }, + "file_paths": "./output/sup3rwind_hourly.nc", + "fwp_chunk_shape": [ + 170, + 170, + 40 + ], + "model_class": "MultiStepGan", + "log_level": "DEBUG", + "log_pattern": "./logs/fwp_temporal_log_{node_index}.log", + "model_kwargs": {"model_dirs": [ + "./sup3rwind_models_202401/step3_1x_12x_14f"]}, + "out_pattern": "./temporal/output/fwp_out_{file_id}.h5", + "spatial_pad": 5, + "temporal_pad": 3, + "exo_kwargs": { + "topography": { + "file_paths": "./output/sup3rwind_hourly.nc", + "res_kwargs": {"combine": "nested", "concat_dim": "Time"}, + "source_file": "PLACEHOLDER", + "exo_resolution": {"spatial": "1km", "temporal": "60min"}, + "steps": [{"model": 0, "combine_type": "input"}]} + } +} diff --git a/examples/sup3rwind/run_configs/wind/config_pipeline.json b/examples/sup3rwind/run_configs/wind/config_pipeline.json new file mode 100755 index 000000000..5cb51c924 --- /dev/null +++ b/examples/sup3rwind/run_configs/wind/config_pipeline.json @@ -0,0 +1,23 @@ +{ + "logging": { + "log_level": "DEBUG" + }, + "pipeline": [ + { + "forward-pass-spatial": "./config_fwp_spatial.json", + "command": "forward-pass" + }, + { + "data-collect-nc": "./config_collect_nc.json", + "command": "data-collect" + }, + { + "forward-pass-temporal": "./config_fwp_temporal.json", + "command": "forward-pass" + }, + { + "data-collect-h5": "./config_collect_h5.json", + "command": "data-collect" + } + ] +} diff --git a/sup3r/postprocessing/file_handling.py b/sup3r/postprocessing/file_handling.py index 7b6ce9c1e..04a9f3b81 100644 --- a/sup3r/postprocessing/file_handling.py +++ b/sup3r/postprocessing/file_handling.py @@ -554,9 +554,8 @@ class OutputHandlerNC(OutputHandler): # pylint: disable=W0613 @classmethod - def _write_output(cls, data, features, lat_lon, times, out_file, - meta_data=None, max_workers=None, gids=None): - """Write forward pass output to NETCDF file + def _get_xr_dset(cls, data, features, lat_lon, times, meta_data=None): + """Convert data to xarray Dataset() object. Parameters ---------- @@ -571,15 +570,8 @@ def _write_output(cls, data, features, lat_lon, times, out_file, Last dimension has ordering (lat, lon) times : pd.Datetimeindex List of times for high res output data - out_file : string - Output file path meta_data : dict | None Dictionary of meta data from model - max_workers : int | None - Has no effect. For compliance with H5 output handler - gids : list - List of coordinate indices used to label each lat lon pair and to - help with spatial chunk data collection """ coords = {'Time': [str(t).encode('utf-8') for t in times], 'south_north': lat_lon[:, 0, 0].astype(np.float32), @@ -595,9 +587,40 @@ def _write_output(cls, data, features, lat_lon, times, out_file, attrs = {k: v if isinstance(v, str) else json.dumps(v) for k, v in meta_data.items()} - with xr.Dataset(data_vars=data_vars, coords=coords, - attrs=attrs) as ncfile: - ncfile.to_netcdf(out_file) + return xr.Dataset(data_vars=data_vars, coords=coords, attrs=attrs) + + # pylint: disable=W0613 + @classmethod + def _write_output(cls, data, features, lat_lon, times, out_file, + meta_data=None, max_workers=None, gids=None): + """Write forward pass output to NETCDF file + + Parameters + ---------- + data : ndarray + (spatial_1, spatial_2, temporal, features) + High resolution forward pass output + features : list + List of feature names corresponding to the last dimension of data + lat_lon : ndarray + Array of high res lat/lon for output data. + (spatial_1, spatial_2, 2) + Last dimension has ordering (lat, lon) + times : pd.Datetimeindex + List of times for high res output data + out_file : string + Output file path + meta_data : dict | None + Dictionary of meta data from model + max_workers : int | None + Has no effect. For compliance with H5 output handler + gids : list + List of coordinate indices used to label each lat lon pair and to + help with spatial chunk data collection + """ + cls._get_xr_dset(data=data, lat_lon=lat_lon, features=features, + times=times, + meta_data=meta_data).to_netcdf(out_file) logger.info(f'Saved output of size {data.shape} to: {out_file}') @classmethod From c7d024f4db6ae8204cfdaf40739508fd76f27bae Mon Sep 17 00:00:00 2001 From: bnb32 Date: Sat, 16 Mar 2024 09:02:58 -0600 Subject: [PATCH 2/7] link to sup3rcc notebook --- examples/sup3rwind/README.rst | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/examples/sup3rwind/README.rst b/examples/sup3rwind/README.rst index bc285db0e..4dfb2be4f 100644 --- a/examples/sup3rwind/README.rst +++ b/examples/sup3rwind/README.rst @@ -7,14 +7,14 @@ Super-Resolution for Renewable Energy Resource Data with Wind from Reanalysis Da Sup3rWind Data Access =================== -The Sup3rWind data is publicly available in a public AWS S3 bucket. The data files can be downloaded directly from there to your local machine or an EC2 instance using the `OEDI data explorer `_ or the `AWS CLI `_. A word of caution: there's a lot of data here. The smallest Sup3rWind file for just a single variable at 2-km 5-minute resolution is 130 GB. +The Sup3rWind data and models are publicly available in a public AWS S3 bucket. The data files can be downloaded directly from there to your local machine or an EC2 instance using the `OEDI data explorer `_ or the `AWS CLI `_. A word of caution: there's a lot of data here. The smallest Sup3rWind file for just a single variable at 2-km 5-minute resolution is 130 GB. The Sup3rWind data is also loaded into `HSDS `_ so that you may stream the data via the `NREL developer API `_ or your own HSDS server. This is the best option if you're not going to want a full annual dataset. See these `rex instructions `_ for more details on how to access this data with HSDS and rex. Example Sup3rWind Data Usage ========================== -Sup3rWind data can be used in generally the same way as Sup3rCC data, with the condition that Sup3rWind includes only wind data and ancillary variables for modeling wind energy generation. Refer to ``/sup3r/examples/sup3rcc/README.rst`` for usage patterns. +Sup3rWind data can be used in generally the same way as Sup3rCC data, with the condition that Sup3rWind includes only wind data and ancillary variables for modeling wind energy generation. Refer to the Sup3rCC example notebook `here `_ for usage patterns. Running Sup3rWind Models ====================== @@ -52,7 +52,7 @@ The Sup3rWind data has versions that coincide with the sup3r software versions. Recommended Citation ==================== -Brandon Benton, Grant Buster, Pavlo Pinchuk, Andrew Glaws, Ryan King, Galen Maclaurin, Ilya Chernyakhovskiy. "Super-Resolution for Renewable Energy Resource Data with Wind from Reanalysis Data (Sup3rWind)". +Brandon Benton, Grant Buster, Pavlo Pinchuk, Andrew Glaws, Ryan King, Galen Maclaurin, Ilya Chernyakhovskiy. "Super-Resolution for Renewable Energy Resource Data with Wind from Reanalysis Data (Sup3rWind)". In Prep. Acknowledgements ================ From 857634e4f428a1f0d22c93d68016426a5a79f327 Mon Sep 17 00:00:00 2001 From: bnb32 Date: Mon, 18 Mar 2024 09:01:26 -0600 Subject: [PATCH 3/7] adding sup3rcc/wind examples to docs --- docs/source/misc/examples.rst | 8 ++++++++ docs/source/misc/examples.sup3rcc.rst | 5 +++++ docs/source/misc/examples.sup3rwind.rst | 5 +++++ 3 files changed, 18 insertions(+) create mode 100644 docs/source/misc/examples.rst create mode 100644 docs/source/misc/examples.sup3rcc.rst create mode 100644 docs/source/misc/examples.sup3rwind.rst diff --git a/docs/source/misc/examples.rst b/docs/source/misc/examples.rst new file mode 100644 index 000000000..96c7a4a79 --- /dev/null +++ b/docs/source/misc/examples.rst @@ -0,0 +1,8 @@ +.. _examples: + +Examples +======== +.. toctree:: + + examples.sup3rcc + examples.sup3rwind \ No newline at end of file diff --git a/docs/source/misc/examples.sup3rcc.rst b/docs/source/misc/examples.sup3rcc.rst new file mode 100644 index 000000000..563db3503 --- /dev/null +++ b/docs/source/misc/examples.sup3rcc.rst @@ -0,0 +1,5 @@ +Sup3rCC Examples +=============== + +.. include:: ../../../examples/sup3rcc/README.rst + :start-line: 2 \ No newline at end of file diff --git a/docs/source/misc/examples.sup3rwind.rst b/docs/source/misc/examples.sup3rwind.rst new file mode 100644 index 000000000..86aee48d0 --- /dev/null +++ b/docs/source/misc/examples.sup3rwind.rst @@ -0,0 +1,5 @@ +Sup3rWind Examples +================= + +.. include:: ../../../examples/sup3rwind/README.rst + :start-line: 2 \ No newline at end of file From 4c2b4ef715cd3f2c0c67446a35f080e262b94c8a Mon Sep 17 00:00:00 2001 From: bnb32 Date: Mon, 18 Mar 2024 09:06:07 -0600 Subject: [PATCH 4/7] requirements update --- requirements.txt | 14 ++++++++++++++ 1 file changed, 14 insertions(+) create mode 100644 requirements.txt diff --git a/requirements.txt b/requirements.txt new file mode 100644 index 000000000..0a9d02624 --- /dev/null +++ b/requirements.txt @@ -0,0 +1,14 @@ +matplotlib>=3.1 +NREL-rex>=0.2.84 +NREL-phygnn>=0.0.23 +NREL-gaps>=0.6.0 +NREL-farms>=1.0.4 +google-auth-oauthlib==0.5.3 +pytest>=5.2 +pillow>=10.0 +tensorflow>2.4,<2.16 +xarray>=2023.0 +netCDF4==1.5.8 +dask>=2022.0 +sphinx>=7.0 +pandas>=2.0 From 212342261a549340556a6700f46ca9b17c98cf84 Mon Sep 17 00:00:00 2001 From: bnb32 Date: Mon, 18 Mar 2024 16:22:30 -0600 Subject: [PATCH 5/7] date modified attr --- sup3r/postprocessing/file_handling.py | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/sup3r/postprocessing/file_handling.py b/sup3r/postprocessing/file_handling.py index 04a9f3b81..4d18a1bd2 100644 --- a/sup3r/postprocessing/file_handling.py +++ b/sup3r/postprocessing/file_handling.py @@ -587,6 +587,10 @@ def _get_xr_dset(cls, data, features, lat_lon, times, meta_data=None): attrs = {k: v if isinstance(v, str) else json.dumps(v) for k, v in meta_data.items()} + attrs['date_modified'] = dt.utcnow().isoformat() + if 'date_created' not in attrs: + attrs['date_created'] = attrs['date_modified'] + return xr.Dataset(data_vars=data_vars, coords=coords, attrs=attrs) # pylint: disable=W0613 From 2f5d4a8bbcaf362092eb07890d17ad2430b54854 Mon Sep 17 00:00:00 2001 From: bnb32 Date: Tue, 19 Mar 2024 06:58:18 -0600 Subject: [PATCH 6/7] missed examples link in index.rst --- docs/source/index.rst | 1 + 1 file changed, 1 insertion(+) diff --git a/docs/source/index.rst b/docs/source/index.rst index 178501e51..21d044d13 100644 --- a/docs/source/index.rst +++ b/docs/source/index.rst @@ -3,6 +3,7 @@ Home page Installation and Usage + Examples API reference <_autosummary/sup3r> CLI reference <_cli/cli> From d985a8009e79491cfae09fe843672805a375a961 Mon Sep 17 00:00:00 2001 From: bnb32 Date: Tue, 19 Mar 2024 07:02:45 -0600 Subject: [PATCH 7/7] separate examples dir for docs --- docs/source/{misc => examples}/examples.rst | 4 ++-- .../{misc/examples.sup3rcc.rst => examples/sup3rcc.rst} | 0 .../{misc/examples.sup3rwind.rst => examples/sup3rwind.rst} | 0 docs/source/index.rst | 2 +- 4 files changed, 3 insertions(+), 3 deletions(-) rename docs/source/{misc => examples}/examples.rst (53%) rename docs/source/{misc/examples.sup3rcc.rst => examples/sup3rcc.rst} (100%) rename docs/source/{misc/examples.sup3rwind.rst => examples/sup3rwind.rst} (100%) diff --git a/docs/source/misc/examples.rst b/docs/source/examples/examples.rst similarity index 53% rename from docs/source/misc/examples.rst rename to docs/source/examples/examples.rst index 96c7a4a79..e53baceec 100644 --- a/docs/source/misc/examples.rst +++ b/docs/source/examples/examples.rst @@ -4,5 +4,5 @@ Examples ======== .. toctree:: - examples.sup3rcc - examples.sup3rwind \ No newline at end of file + sup3rcc + sup3rwind \ No newline at end of file diff --git a/docs/source/misc/examples.sup3rcc.rst b/docs/source/examples/sup3rcc.rst similarity index 100% rename from docs/source/misc/examples.sup3rcc.rst rename to docs/source/examples/sup3rcc.rst diff --git a/docs/source/misc/examples.sup3rwind.rst b/docs/source/examples/sup3rwind.rst similarity index 100% rename from docs/source/misc/examples.sup3rwind.rst rename to docs/source/examples/sup3rwind.rst diff --git a/docs/source/index.rst b/docs/source/index.rst index 21d044d13..6b7d5b0aa 100644 --- a/docs/source/index.rst +++ b/docs/source/index.rst @@ -3,7 +3,7 @@ Home page Installation and Usage - Examples + Examples API reference <_autosummary/sup3r> CLI reference <_cli/cli>