Daily Temporal Climatology using xcdat #469
Replies: 4 comments 15 replies
-
I would recommend checking if you can get the results you need with smaller datasets, or rewriting your script and algorithm to loop on smaller subsets of the data On Linux, you could also try to use cdo to reduce your data before loading it in your python script. They seem to provide some Win binaries that you could try to use |
Beta Was this translation helpful? Give feedback.
-
I have this problem a lot.
If it was me, I would divide up the calculation spatially. If latitude-longitude data, I would load one latitude line in at a time via a loop.
Michael
… On May 10, 2023, at 2:19 AM, Jean-Yves Peterschmitt ***@***.***> wrote:
117. GiB is indeed quite big, and would be a problem even on a Linux cluster!
I would recommend checking if you can get the results you need with smaller datasets, or rewriting your script and algorithm to loop on smaller subsets of the data
On Linux, you could also try to use cdo <https://code.mpimet.mpg.de/projects/cdo> to reduce your data before loading it in your python script. They seem to provide some Win binaries <https://code.mpimet.mpg.de/projects/cdo/files> that you could try to use
—
Reply to this email directly, view it on GitHub <#469 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ACSKCHGX57VKIXNA75GH7V3XFNMTHANCNFSM6AAAAAAXYVWWGA>.
You are receiving this because you are subscribed to this thread.
|
Beta Was this translation helpful? Give feedback.
-
I'm trying to replicate the seasonal climatologies as https://github.com/xCDAT/xcdat/blob/main/docs/examples/climatology-and-departures.ipynb
For seasonal climatology: I tried the follwoing dsc_seas = dsc.temporal.climatology(
"icec",
freq="season",
weighted=True,
season_config={"dec_mode": "DJF", "drop_incomplete_djf": True},
) which returns the follwoing error:
My seasonal climatology data (given below) although returned 4 time variables (4 seasons), while plotting displays the Runtime Warning with blank plots
My Goal: Seasonal nd Daily anomalies using the xcdat temporal climatology. |
Beta Was this translation helpful? Give feedback.
-
That would be great, seriously.
Climate data generally deals with such huge chunks, but xcdat provides
easier and accurate solutions.
Exploring other API too from the module.
Thanks alot for the response again.
*Girija Kalyani BuradaDoctoral StudentSchool of Geography, Environment and
Earth SciencesVictoria University of WellingtonNew Zealand*
…On Thu, May 11, 2023 at 6:45 PM Stephen Po-Chedley ***@***.***> wrote:
That is great news – thank you for following up. I was confused, because
these are *warnings* (not *errors*) so it seems like the calculation is
(or should be) completing. I think there may be lingering memory issues
since this dataset is so large – this may be why the restart helped. I
think we're going to explore xcdat + dask in the future, so we may add more
documentation about this. We'll keep in mind large datasets as something to
contend with.
—
Reply to this email directly, view it on GitHub
<#469 (reply in thread)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/ABWDHEHYS2R5OIR6BJSNCATXFSDJRANCNFSM6AAAAAAXYVWWGA>
.
You are receiving this because you modified the open/close state.Message
ID: ***@***.***>
|
Beta Was this translation helpful? Give feedback.
-
Hi there,
I have daily data for about 40 years, each year have been imported using mf_datset and nested along 'time' dimension.
Trying to use Daily temporal climatology on this, and it shows memory issues.
The error reads as follows:
Can you please provide insights on how to handle it?
Many Thanks in advance.
![image](https://user-images.githubusercontent.com/7091088/236663969-02137ce6-97da-4b22-8ce9-1d97553c3d0f.png)
Also, how can I apply dask chunks on this.
My data looks like this:
Beta Was this translation helpful? Give feedback.
All reactions