-
Notifications
You must be signed in to change notification settings - Fork 164
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
pysteps 2.0 #216
Comments
At this point, it would be interesting to hear the opinion of the other core contributors @pulkkins @aperezhortal @cvelascof @RubenImhoff and in general the whole pysteps community. Also, the list above is by no means exhaustive nor fixed. Version 2 will be the opportunity to introduce substantial changes based on our experience so far: we need your inputs! For this purpose, I created a new project where to list and track issues and PRs related to this effort. In terms of roadmap, I'd suggest the following: Phase 1: up to end of October 2021(*):
Goal: pysteps-v2 is published as a pre-release (beta) Phase 2: up to September 2022 (ERAD):
Goal: consolidate v2, which becomes default version. Phase 3:
Goal: close project, publish results. (*) This deadline is given by time constrains on the RMI side, which has the lead on this initial phase of the project (@ladc ) |
Hi everyone, Unfortunately I couldn't join the last meeting due to a week of vacation, but this looks really good! It is great to see how our initial idea to pick up blending again is a reason to move forward to a v2 of pysteps! The present ideas are what I was thinking of as well, so not much to add at this point. I think that during my stay at RMI in August, we will come up with some (additional) ideas. The new importer that allows for importing radar and NWP data, hopefully allowing for some sort of up-/downscaling in space and time to match the resolution of the radar and NWP data, will be one of the challenges here. I think if we can make that as generic as possible, we already have a super useful tool (let alone for the blending and other functionalities that we will add). So, a great move forward! In a week from now, I can devote most of my time to the project, so we will keep in touch! |
Hi everyone, Version 2 sounds great 💪 This new version maybe is a good opportunity to change the pystepsrc parameter file format. Although the JSON configuration file does the job, it has a few disadvantages. A better, and more readable, alternative is the YAML format that I personally like a lot. Also, this is a good opportunity to revisit the interface as @ladc mentioned in point 4. For example, using an interface like Scikit-learn allows a better integration of PySTEPS with other ML libraries (pytorch, keras, scikit-learn, etc.). |
Hi everyone, I was also on a vacation, so I could not join the last meeting. The points listed by @dnerini are very good and clearly show the limitations of the current version. It should be noted that its main goal was to implement a Python version of STEPS. As a result, the codebase is not generic enough to contain all the planned features (or even the currently implemented ones). The design was also a bit rushed since we needed a minimal working version for ERAD 2018. For implementing more advanced features like blending (item 1), a new data model (item 2) or a more generic interface compatible with machine learning methods (item 4), a major redesign is the only way to go. |
Hi everyone, I say this for all those like me who find themselves installing libraries in AWS Lambda (and similar) where space is precious and therefore it is very convenient to be able to install only what you need without having to manually delete files and folders after installation. |
Hi @fox91, thanks for the suggestion. I like the idea. I did a quick search and it seems that maybe we can do that using something like Namespace packages. |
I don't know how is implemented but I like the way I can install submodules in |
Hi @fox91 In a way, we already partially support the same idea by using optional dependencies. For example, This should be easily achievable with the extra_requires option in the setup.cfg file. Or do you have something different in mind? (ps: I suggest we open a dedicated issue and continue the discussion there?) |
I love the idea of using xarray for pySTEPSv2.
What are your thoughts? |
All very relevant points, thanks @cvelascof for raising them. This is my personal opinion: I see this as an iterative process, where version 2.0.0 should include the changes necessary to provide support to xarray across the whole library. This means that all methods need to be able to use and return xarray objects instead of numpy arrays as it is currently done. meaning:
How this changes will be implemented in practice will depend a lot on the individual methods. My impressions is that most methods would benefit from a more in-depth refactoring leveraging the advanced functionalities of xarray, but I'm also conscious that this would represent a considerable effort. As a compromise, we could adapt the most complex methods (as for example the existing nowcasting methods) so that the input xarray object is simply converted into a numpy array at the beginning of the function, and back to an xarray at the very end (I guess this would correspond more or less to the wrapping option that you mention above). Because the switch to v2, we can introduce breaking changes, including renaming methods and modules. So I wouldn't worry too much in terms of back-compatibility, although it makes sense to try and keep the general structure as close as possible to v1. There will be plenty of details to sort out, but I think we can discuss and solve them within the relevant PRs as we proceed with the refactoring (see #223 for a practical example). Edit: just to make it clearer, I don't think we want to create new versions of each routine (your option 2), that would just be very confusing. Personally, I see pysteps V2 working with xarray objects only (at least in terms of interfaces). This means discarding your options 1 and 2, and going for option 3 for methods that are too expensive to refactor, while applying a more in-depth refactoring where possible. What do you think? |
During a meeting with @dnerini, @loforest and @wdewettin, the suggestion was made to start working towards a new version of pysteps.
The text was updated successfully, but these errors were encountered: