multiple variables in a signle grib file successfully loaded into dataset and dataframe #290
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
when more then one variable found in grib file and conflicts with existing keys like "time" and "step" we will add special identifier to that key with _NumberSequence When file containing multiple variables it might possible that we have multiple grib versions in a same file to handle it we are combining those values in a single string.
|
|
|
Hi @alexamici, can you review this PR which fixes issue for loading grib file with multiple weather variables. |
|
Hi @vasarabharat, testing your branch on my local data does not give similar results to you (i.e. GRIB file with multiple variables on different steps still does not load) - could you provide the request you used to get data from the CDS, or else post a link to a GRIB file that illustrates the problem? Thanks! It may be that this is not the preferred solution to the issue, but I'd like to assess it better. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Trying to get multiple parameters in a single grib file using cds api, but failed to load in dataset because of following errors, applied some fixes as describe bellow.
When more then one variable found in grib file and conflicts with existing keys like "time" and "step" etc,. we will add special identifier to that key with _NumberSequence
The Error That will be resolved by this fix:
Traceback (most recent call last):
File "/home/bharat/.local/lib/python3.8/site-packages/cfgrib/dataset.py", line 660, in build_dataset_components
dict_merge(variables, coord_vars)
File "/home/bharat/.local/lib/python3.8/site-packages/cfgrib/dataset.py", line 591, in dict_merge
raise DatasetBuildError(
cfgrib.dataset.DatasetBuildError: key present and new value is different: key='time' value=Variable(dimensions=(), data=1640944800) new_value=Variable(dimensions=(), data=1640908800)
output dataset will have time, time_0, time_1 as a identifier names.
When file containing multiple variables it might possible that we have multiple grib versions in a same file to handle it we are combining those values in a single string.
The Error That will be resolved by this fix:
Traceback (most recent call last):
File "gribtest.py", line 9, in
ds = xr.open_dataset(gribFileName, engine="cfgrib")
File "/home/bharat/Documents/GitHub/xarray/xarray/backends/api.py", line 495, in open_dataset
backend_ds = backend.open_dataset(
File "/home/bharat/.local/lib/python3.8/site-packages/cfgrib/xarray_plugin.py", line 99, in open_dataset
store = CfGribDataStore(
File "/home/bharat/.local/lib/python3.8/site-packages/cfgrib/xarray_plugin.py", line 39, in init
self.ds = opener(filename, **backend_kwargs)
File "/home/bharat/.local/lib/python3.8/site-packages/cfgrib/dataset.py", line 764, in open_file
return open_from_index(index, read_keys, time_dims, extra_coords, **kwargs)
File "/home/bharat/.local/lib/python3.8/site-packages/cfgrib/dataset.py", line 706, in open_from_index
dimensions, variables, attributes, encoding = build_dataset_components(
File "/home/bharat/.local/lib/python3.8/site-packages/cfgrib/dataset.py", line 675, in build_dataset_components
attributes = build_dataset_attributes(index, filter_by_keys, encoding)
File "/home/bharat/.local/lib/python3.8/site-packages/cfgrib/dataset.py", line 599, in build_dataset_attributes
attributes = enforce_unique_attributes(index, GLOBAL_ATTRIBUTES_KEYS, filter_by_keys)
File "/home/bharat/.local/lib/python3.8/site-packages/cfgrib/dataset.py", line 273, in enforce_unique_attributes
raise DatasetBuildError("multiple values for key %r" % key, key, fbks)
cfgrib.dataset.DatasetBuildError: multiple values for key 'edition'
Final Output dataset will looks like this (downloaded all variables for 2018-Jan-01 10:00):
Dimensions: (latitude: 1801, longitude: 3600)
Coordinates:
number int64 ...
time datetime64[ns] ...
step timedelta64[ns] ...
surface float64 ...
valid_time datetime64[ns] ...
depthBelowLandLayer float64 ...
Data variables: (12/98)
u10 (latitude, longitude) float32 ...
v10 (latitude, longitude) float32 ...
d2m (latitude, longitude) float32 ...
t2m (latitude, longitude) float32 ...
evabs (latitude, longitude) float32 ...
evaow (latitude, longitude) float32 ...
... ...
depthBelowLandLayer_4 float64 ...
swvl3 (latitude, longitude) float32 ...
time_20 datetime64[ns] ...
step_20 timedelta64[ns] ...
depthBelowLandLayer_5 float64 ...
swvl4 (latitude, longitude) float32 ...
Attributes:
GRIB_edition: 1 AND 2
GRIB_centre: ecmf
GRIB_centreDescription: European Centre for Medium-Range Weather Forecasts
GRIB_subCentre: 0
Conventions: CF-1.7
institution: European Centre for Medium-Range Weather Forecasts
history: 2022-03-03T15:23 GRIB to CDM+CF via cfgrib-0.9.1...
if we convert this into dataframe the then it will looks like
latitude longitude number time step surface
0 90.0 0.0 0 2018-01-01 0 days 10:00:00 0.0
1 90.0 0.1 0 2018-01-01 0 days 10:00:00 0.0
2 90.0 0.2 0 2018-01-01 0 days 10:00:00 0.0
3 90.0 0.3 0 2018-01-01 0 days 10:00:00 0.0
4 90.0 0.4 0 2018-01-01 0 days 10:00:00 0.0
... ... ... ... ... ... ...
6483595 -90.0 359.5 0 2018-01-01 0 days 10:00:00 0.0
6483596 -90.0 359.6 0 2018-01-01 0 days 10:00:00 0.0
6483597 -90.0 359.7 0 2018-01-01 0 days 10:00:00 0.0
6483598 -90.0 359.8 0 2018-01-01 0 days 10:00:00 0.0
6483599 -90.0 359.9 0 2018-01-01 0 days 10:00:00 0.0
0 2018-01-01 10:00:00 NaN NaN NaN NaN
1 2018-01-01 10:00:00 NaN NaN NaN NaN
2 2018-01-01 10:00:00 NaN NaN NaN NaN
3 2018-01-01 10:00:00 NaN NaN NaN NaN
4 2018-01-01 10:00:00 NaN NaN NaN NaN
... ... ... ... ... ...
6483595 2018-01-01 10:00:00 -6.011181 1.568486 244.020462 247.884399
6483596 2018-01-01 10:00:00 -6.011181 1.568486 244.020462 247.884399
6483597 2018-01-01 10:00:00 -6.011181 1.568486 244.020462 247.884399
6483598 2018-01-01 10:00:00 -6.011181 1.568486 244.020462 247.884399
6483599 2018-01-01 10:00:00 -6.011181 1.568486 244.020462 247.884399
0 NaN NaN NaN NaN NaN 2018-01-01 10:00:00 0 days
1 NaN NaN NaN NaN NaN 2018-01-01 10:00:00 0 days
2 NaN NaN NaN NaN NaN 2018-01-01 10:00:00 0 days
3 NaN NaN NaN NaN NaN 2018-01-01 10:00:00 0 days
4 NaN NaN NaN NaN NaN 2018-01-01 10:00:00 0 days
... ... ... ... ... ... ... ...
6483595 0.0 0.0 0.0 0.0 0.85 2018-01-01 10:00:00 0 days
6483596 0.0 0.0 0.0 0.0 0.85 2018-01-01 10:00:00 0 days
6483597 0.0 0.0 0.0 0.0 0.85 2018-01-01 10:00:00 0 days
6483598 0.0 0.0 0.0 0.0 0.85 2018-01-01 10:00:00 0 days
6483599 0.0 0.0 0.0 0.0 0.85 2018-01-01 10:00:00 0 days
0 NaN 2018-01-01 10:00:00 0 days NaN 2018-01-01 10:00:00
1 NaN 2018-01-01 10:00:00 0 days NaN 2018-01-01 10:00:00
2 NaN 2018-01-01 10:00:00 0 days NaN 2018-01-01 10:00:00
3 NaN 2018-01-01 10:00:00 0 days NaN 2018-01-01 10:00:00
4 NaN 2018-01-01 10:00:00 0 days NaN 2018-01-01 10:00:00
... ... ... ... ... ...
6483595 277.130127 2018-01-01 10:00:00 0 days 3.0 2018-01-01 10:00:00
6483596 277.130127 2018-01-01 10:00:00 0 days 3.0 2018-01-01 10:00:00
6483597 277.130127 2018-01-01 10:00:00 0 days 3.0 2018-01-01 10:00:00
6483598 277.130127 2018-01-01 10:00:00 0 days 3.0 2018-01-01 10:00:00
6483599 277.130127 2018-01-01 10:00:00 0 days 3.0 2018-01-01 10:00:00
0 0 days NaN 2018-01-01 10:00:00 0 days NaN
1 0 days NaN 2018-01-01 10:00:00 0 days NaN
2 0 days NaN 2018-01-01 10:00:00 0 days NaN
3 0 days NaN 2018-01-01 10:00:00 0 days NaN
4 0 days NaN 2018-01-01 10:00:00 0 days NaN
... ... ... ... ... ...
6483595 0 days 251.126831 2018-01-01 10:00:00 0 days 0.009766
6483596 0 days 251.126831 2018-01-01 10:00:00 0 days 0.009766
6483597 0 days 251.126831 2018-01-01 10:00:00 0 days 0.009766
6483598 0 days 251.126831 2018-01-01 10:00:00 0 days 0.009766
6483599 0 days 251.126831 2018-01-01 10:00:00 0 days 0.009766
0 2018-01-01 10:00:00 0 days NaN 2018-01-01 10:00:00 0 days
1 2018-01-01 10:00:00 0 days NaN 2018-01-01 10:00:00 0 days
2 2018-01-01 10:00:00 0 days NaN 2018-01-01 10:00:00 0 days
3 2018-01-01 10:00:00 0 days NaN 2018-01-01 10:00:00 0 days
4 2018-01-01 10:00:00 0 days NaN 2018-01-01 10:00:00 0 days
Continue..