Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Multiple Zarr to Rule them All #54

Closed
wants to merge 67 commits into from
Closed

Multiple Zarr to Rule them All #54

wants to merge 67 commits into from

Commits on May 5, 2024

  1. yaml_config for cosmo data

    Simon Adamov committed May 5, 2024
    Configuration menu
    Copy the full SHA
    e5c245f View commit details
    Browse the repository at this point in the history

  2. initial version of single zarr dataset

    Simon Adamov committed May 5, 2024
    Configuration menu
    Copy the full SHA
    33e7ecf View commit details
    Browse the repository at this point in the history

Commits on May 6, 2024

  1. handling None zarrs

    Simon Adamov committed May 6, 2024
    Configuration menu
    Copy the full SHA
    9936e3b View commit details
    Browse the repository at this point in the history

Commits on May 8, 2024

  1. removed all dependencies on constants.py 

    user configs are retrieved either from data_config.yaml
or they are set as flags to train_model.py
Capabilities of ConfigLoader class extended
    Simon Adamov committed May 8, 2024
    Configuration menu
    Copy the full SHA
    774d16a View commit details
    Browse the repository at this point in the history

  2. fix linter

    Simon Adamov committed May 8, 2024
    Configuration menu
    Copy the full SHA
    7bb139b View commit details
    Browse the repository at this point in the history

  3. Fixed calls to new WeatherDataModule Class

    Simon Adamov committed May 8, 2024
    Configuration menu
    Copy the full SHA
    af076fe View commit details
    Browse the repository at this point in the history

  4. fix linter

    Simon Adamov committed May 8, 2024
    Configuration menu
    Copy the full SHA
    147caec View commit details
    Browse the repository at this point in the history

  5. upload data config to wandb for history logs

    Simon Adamov committed May 8, 2024
    Configuration menu
    Copy the full SHA
    2b65416 View commit details
    Browse the repository at this point in the history

  6. Improved handling of static data

    Simon Adamov committed May 8, 2024
    Configuration menu
    Copy the full SHA
    ed9ed69 View commit details
    Browse the repository at this point in the history

  7. dask and zarr are required backends to xarray

    Simon Adamov committed May 8, 2024
    Configuration menu
    Copy the full SHA
    0b69f4e View commit details
    Browse the repository at this point in the history

Commits on May 9, 2024

  1. Implements windowed forcing and boundary

    Simon Adamov committed May 9, 2024
    Configuration menu
    Copy the full SHA
    b76d078 View commit details
    Browse the repository at this point in the history

  2. Some project related stuff (simple setup to pip install -e .)

    Simon Adamov committed May 9, 2024
    Configuration menu
    Copy the full SHA
    5d27a4c View commit details
    Browse the repository at this point in the history

  3. introducing realistic boundaries

    Simon Adamov committed May 9, 2024
    Configuration menu
    Copy the full SHA
    4dadf29 View commit details
    Browse the repository at this point in the history

  4. Adapted nwp_xy related code to new data loading procedure

    Simon Adamov committed May 9, 2024
    Configuration menu
    Copy the full SHA
    7524c4d View commit details
    Browse the repository at this point in the history

  5. only state requires units for plotting 

    lat lon specifications make the code more flexible
    Simon Adamov committed May 9, 2024
    Configuration menu
    Copy the full SHA
    45fd375 View commit details
    Browse the repository at this point in the history

  6. small bugfixes and improvements

    Simon Adamov committed May 9, 2024
    Configuration menu
    Copy the full SHA
    812323d View commit details
    Browse the repository at this point in the history

  7. Calculate stats and store in zarr archive 

    Zarr is registered to model buffer
Normalization happens on device on_after_batch_transfer
    Simon Adamov committed May 9, 2024
    Configuration menu
    Copy the full SHA
    500f2fb View commit details
    Browse the repository at this point in the history

  8. latex support

    Simon Adamov committed May 9, 2024
    Configuration menu
    Copy the full SHA
    9293fe1 View commit details
    Browse the repository at this point in the history

  9. ar_steps for training and eval

    Simon Adamov committed May 9, 2024
    Configuration menu
    Copy the full SHA
    e80aa58 View commit details
    Browse the repository at this point in the history

  10. smaller ammendments

    Simon Adamov committed May 9, 2024
    Configuration menu
    Copy the full SHA
    a86fc07 View commit details
    Browse the repository at this point in the history

  11. Dummy mask was inverted - fixed

    Simon Adamov committed May 9, 2024
    Configuration menu
    Copy the full SHA
    7ae9c87 View commit details
    Browse the repository at this point in the history

  12. replace hardcoded normalization path

    Simon Adamov committed May 9, 2024
    Configuration menu
    Copy the full SHA
    93674a2 View commit details
    Browse the repository at this point in the history

Commits on May 23, 2024

  1. create 3D mesh objects for schol AR

    Simon Adamov committed May 23, 2024
    Configuration menu
    Copy the full SHA
    f2cbc44 View commit details
    Browse the repository at this point in the history

Commits on May 28, 2024

  1. fixed math writing

    @sadamov
    sadamov committed May 28, 2024
    Configuration menu
    Copy the full SHA
    c0e7529 View commit details
    Browse the repository at this point in the history

  2. Merge branch 'feature_dataset_yaml' of https://github.com/joeloskarss… 

    …on/neural-lam into feature_dataset_yaml
    @sadamov
    sadamov committed May 28, 2024
    Configuration menu
    Copy the full SHA
    af7751a View commit details
    Browse the repository at this point in the history

  3. cherry-pick with main

    @sadamov
    sadamov committed May 28, 2024
    Configuration menu
    Copy the full SHA
    5f538f9 View commit details
    Browse the repository at this point in the history

  4. bugfixes

    @sadamov
    sadamov committed May 28, 2024
    Configuration menu
    Copy the full SHA
    6685e94 View commit details
    Browse the repository at this point in the history

  5. pre_commits

    @sadamov
    sadamov committed May 28, 2024
    Configuration menu
    Copy the full SHA
    6423fdf View commit details
    Browse the repository at this point in the history

Commits on May 31, 2024

  1. Merge remote-tracking branch 'origin/main' into feature_dataset_yaml

    @sadamov
    sadamov committed May 31, 2024
    Configuration menu
    Copy the full SHA
    59c4947 View commit details
    Browse the repository at this point in the history

  2. config.py is ready for danra

    @sadamov
    sadamov committed May 31, 2024
    Configuration menu
    Copy the full SHA
    4e457ed View commit details
    Browse the repository at this point in the history

Commits on Jun 1, 2024

  1. streamlined multi-zarr workflow

    @sadamov
    sadamov committed Jun 1, 2024
    Configuration menu
    Copy the full SHA
    adc592f View commit details
    Browse the repository at this point in the history

  2. xarray zarr based data normalization

    @sadamov
    sadamov committed Jun 1, 2024
    Configuration menu
    Copy the full SHA
    a7bea6b View commit details
    Browse the repository at this point in the history

Commits on Jun 2, 2024

  1. adjusted pre-processing scripts to new data config workflow

    @sadamov
    sadamov committed Jun 2, 2024
    Configuration menu
    Copy the full SHA
    1f7cbe8 View commit details
    Browse the repository at this point in the history

  2. plotting update with latest get_xy() function

    @sadamov
    sadamov committed Jun 2, 2024
    Configuration menu
    Copy the full SHA
    e328152 View commit details
    Browse the repository at this point in the history

  3. making data config more modular

    @sadamov
    sadamov committed Jun 2, 2024
    Configuration menu
    Copy the full SHA
    cb85cda View commit details
    Browse the repository at this point in the history

  4. removing boundaries for now

    @sadamov
    sadamov committed Jun 2, 2024
    Configuration menu
    Copy the full SHA
    eb8c6fb View commit details
    Browse the repository at this point in the history

  5. small updates

    @sadamov
    sadamov committed Jun 2, 2024
    Configuration menu
    Copy the full SHA
    0cfbb33 View commit details
    Browse the repository at this point in the history

  6. improved stats and units retrieval

    @sadamov
    sadamov committed Jun 2, 2024
    Configuration menu
    Copy the full SHA
    59d0c8a View commit details
    Browse the repository at this point in the history

Commits on Jun 3, 2024

  1. improved zarr-based normalization

    @sadamov
    sadamov committed Jun 3, 2024
    Configuration menu
    Copy the full SHA
    668dd81 View commit details
    Browse the repository at this point in the history

  2. merge main

    @sadamov
    sadamov committed Jun 3, 2024
    Configuration menu
    Copy the full SHA
    374d032 View commit details
    Browse the repository at this point in the history

  3. Introduced datetime forcing calculation as seperate script

    @sadamov
    sadamov committed Jun 3, 2024
    Configuration menu
    Copy the full SHA
    8fa3ca7 View commit details
    Browse the repository at this point in the history

  4. Fixed order of y and x dims to adhere to #52

    @sadamov
    sadamov committed Jun 3, 2024
    Configuration menu
    Copy the full SHA
    a748903 View commit details
    Browse the repository at this point in the history

Commits on Jun 4, 2024

  1. cleanup: boundary_mask, zarr-opening, utils

    @sadamov
    sadamov committed Jun 4, 2024
    Configuration menu
    Copy the full SHA
    74b4a10 View commit details
    Browse the repository at this point in the history

  2. Merge remote-tracking branch 'origin/main' into feature_dataset_yaml

    @sadamov
    sadamov committed Jun 4, 2024
    Configuration menu
    Copy the full SHA
    0a041d1 View commit details
    Browse the repository at this point in the history

  3. adapted tests for zarr-analysis data

    @sadamov
    sadamov committed Jun 4, 2024
    Configuration menu
    Copy the full SHA
    425123c View commit details
    Browse the repository at this point in the history

  4. Readme adapted for yaml zarr analysis workflow

    @sadamov
    sadamov committed Jun 4, 2024
    Configuration menu
    Copy the full SHA
    4dcf671 View commit details
    Browse the repository at this point in the history

  5. samller bugfixes and improvements

    @sadamov
    sadamov committed Jun 4, 2024
    Configuration menu
    Copy the full SHA
    6d384f0 View commit details
    Browse the repository at this point in the history

  6. Added fixed data config file for testing on Danra

    @sadamov
    sadamov committed Jun 4, 2024
    Configuration menu
    Copy the full SHA
    12ff4f2 View commit details
    Browse the repository at this point in the history

  7. reducing runtime of tests with smaller sample

    @sadamov
    sadamov committed Jun 4, 2024
    Configuration menu
    Copy the full SHA
    03f7769 View commit details
    Browse the repository at this point in the history

Commits on Jun 6, 2024

  1. download danra data for test and example (streaming not possible)

    @sadamov
    sadamov committed Jun 6, 2024
    Configuration menu
    Copy the full SHA
    26f069c View commit details
    Browse the repository at this point in the history

  2. bugfixes after real-life testcase

    @sadamov
    sadamov committed Jun 6, 2024
    Configuration menu
    Copy the full SHA
    1f1cbcc View commit details
    Browse the repository at this point in the history

  3. Merge remote-tracking branch 'origin/main' into feature_dataset_yaml

    @sadamov
    sadamov committed Jun 6, 2024
    Configuration menu
    Copy the full SHA
    b369306 View commit details
    Browse the repository at this point in the history

  4. organize .zarr in /data

    @sadamov
    sadamov committed Jun 6, 2024
    Configuration menu
    Copy the full SHA
    0cdc361 View commit details
    Browse the repository at this point in the history

  5. cleanup

    @sadamov
    sadamov committed Jun 6, 2024
    Configuration menu
    Copy the full SHA
    23ca7b3 View commit details
    Browse the repository at this point in the history

  6. linter

    @sadamov
    sadamov committed Jun 6, 2024
    Configuration menu
    Copy the full SHA
    81422f1 View commit details
    Browse the repository at this point in the history

Commits on Jun 7, 2024

  1. static dataset doesn't have time dim

    @sadamov
    sadamov committed Jun 7, 2024
    Configuration menu
    Copy the full SHA
    124541b View commit details
    Browse the repository at this point in the history

  2. making two complex functions more modular

    @sadamov
    sadamov committed Jun 7, 2024
    Configuration menu
    Copy the full SHA
    6140fdb View commit details
    Browse the repository at this point in the history

Commits on Jun 8, 2024

  1. chunk dataset by time

    @sadamov
    sadamov committed Jun 8, 2024
    Configuration menu
    Copy the full SHA
    db6a912 View commit details
    Browse the repository at this point in the history

  2. create list first for performance

    @sadamov
    sadamov committed Jun 8, 2024
    Configuration menu
    Copy the full SHA
    1aaa8dc View commit details
    Browse the repository at this point in the history

  3. converting to_array is very slow 

    the behavior of xr.Dataset.dims will change soon
    @sadamov
    sadamov committed Jun 8, 2024
    Configuration menu
    Copy the full SHA
    81856b2 View commit details
    Browse the repository at this point in the history

  4. allow for forcings to not be normalized

    @sadamov
    sadamov committed Jun 8, 2024
    Configuration menu
    Copy the full SHA
    b3da818 View commit details
    Browse the repository at this point in the history

  5. allow non_normalized_vars to be null

    @sadamov
    sadamov committed Jun 8, 2024
    Configuration menu
    Copy the full SHA
    7ee5398 View commit details
    Browse the repository at this point in the history

  6. fixed coastlines using new xy_extent function

    @sadamov
    sadamov committed Jun 8, 2024
    Configuration menu
    Copy the full SHA
    4782103 View commit details
    Browse the repository at this point in the history

Commits on Jun 9, 2024

  1. Some projections return inverted axes (rotatedPole)

    @sadamov
    sadamov committed Jun 9, 2024
    Configuration menu
    Copy the full SHA
    e0ffc5b View commit details
    Browse the repository at this point in the history

Commits on Jun 13, 2024

  1. Docstrings added

    @sadamov
    sadamov committed Jun 13, 2024
    Configuration menu
    Copy the full SHA
    c1f43b7 View commit details
    Browse the repository at this point in the history

Commits on Aug 7, 2024

  1. Merge branch 'main' into feature_dataset_yaml

    @sadamov
    sadamov committed Aug 7, 2024
    Configuration menu
    Copy the full SHA
    1a1f5b2 View commit details
    Browse the repository at this point in the history

  2. Bugfix training without forcing data

    @sadamov
    sadamov committed Aug 7, 2024
    Configuration menu
    Copy the full SHA
    5f72fe3 View commit details
    Browse the repository at this point in the history