Dynamic date field for ERA5T query problems

[khufkens]

Sooooo.... it seems that the CDS API has an undocumented (or at least not well advertised) feature which allows you to specify date ranges (+ hours) instead of the year / month / day / hour combos that the CDS webpage suggests.

this would allow one to be way more granular in their queries, AND avoid the bug that is hunting the current API when it comes to the ERA5T mix up and data unavailability (see top part link below).

https://confluence.ecmwf.int/pages/viewpage.action?pageId=277352608&focusedCommentId=278541955#comment-278541955

Will have to test this, but if as suggested, this works this should be integrated. A workaround for the ERA5T bug would be to capture the date range from the year / month / day fields and reformat the string during API to request conversion (if full sequential coverage is requested). The latter would sidestep the bug described BUT will generate a HUGE query (all dates are listed). Ok scrap the last remark, seemingly there is another option using the old MARS format (or similar), i.e. 2022-06-01/2022-06-06 to specify the date range.

Lots to be considered, @eliocamp input always appreciated.

[eliocamp]

I don't know how this works, really, so I don't feel I can give any specific insight. In general, if it's not documented, I'd assume that it's not for public consumption and could break at any time, so I'd be wary of supporting it.

[khufkens]

Tend to agree, but the current API fails badly as shown by many. So this is rather a back stop for ECMWF not resolving the issue. This has impacted my work and that of others (if not severely due to unnecessary juggling of files) so if ECMWF doesn't provide a fix I fear that I'll have to do so. With the odd chance that it will break later on.

[khufkens]

Did some testing and the date = "YYYY-MM-DD/YYYY-MM-DD" syntax works. However, this feature is actively hidden by ECMWF as by censoring of the linked issue (see my note here:
https://twitter.com/koen_hufkens/status/1539586220250071042). Officially this only leaves you with querying data year by year or everything up to the last year and then doing an extra pass for the last year. Poor performance of ECMWF on this one I must admit.

[eliocamp]

Ok, so now I'm on a project that requires me to get up-to-date data on a regular basis so I'm testing it with:

request <- list(product_type = "reanalysis", 
                format = "netcdf", 
                variable = "geopotential", 
                dataset_short_name = "reanalysis-era5-pressure-levels",
                pressure_level = c("700", "50"),
                time = "00:00", 
                area = c(-20, 0, -90, 360),
                grid = c(2.5, 2.5),
                target = "download.nc", 
                date = "1959-01-01/2022-06-30")

The request hasn't been rejected and has been running for 12 minutes (and I'm bored while waiting, hence this comment 🤣).

Assuming this works, my question is what would need to change in ecmwfr?

[khufkens]

It works, I've been using it for some of my ornithology work.

The thing that needs to change is the python rstudio plugin I think, to automatically translate the original query to a universal date query. That's it. This would be a universal fix. But note that ECMWF doesn't like us doing this. In the issue on the forum they at times explicitly removed references to it!!!

[khufkens]

What I mentioned above is the reason why I've been reluctant to do anything, although ideal. For example for large queries they suggest to send queries by month, not as a whole chunk etc. Not sure to what extend we should / could cheat on this. As long as the user base is small enough this is not a problem, but if it grows we might get slapped or worse the date feature is removed.

[eliocamp]

I wouldn't translate year/month/day queries to date/date by default since it's non-standard and discouraged and it only makes a difference in the specific case of retrieving the latest data.

I'd say that if the user can use the date query in ecmwfr, then it's as supported as it can be. Users that need and know to use it will probably be comfortable enough with the syntax to modify the request themselves.

Curiously, I tried to search for documentation about the per-month queries and found this article that actually uses the date syntax.
https://confluence.ecmwf.int/display/CKB/Atmosphere+Data+Store+%28ADS%29+documentation#AtmosphereDataStore(ADS)documentation-Efficiencytips

[khufkens]

Ok, now I'm confused. Why would they hide it and discourage it?

Anyway. I agree, that to provide an option that doesn't rub anyone the wrong way we could write an issue with instructions and just pin it on top of the package issue list. In addition, some mention in the vignettes can help sort things out (while referencing to the original docs to get under liability etc).

[eliocamp]

I'm looking at the readme and it already includes the date syntax xD

[khufkens]

Always good to find out that the issue is already fixed.