Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Break up processing into chunks #68

Open
gutmann opened this issue Jun 8, 2018 · 0 comments
Open

Break up processing into chunks #68

gutmann opened this issue Jun 8, 2018 · 0 comments

Comments

@gutmann
Copy link
Member

gutmann commented Jun 8, 2018

To permit larger runs in limited memory it would be nice to be able to specify a chunk size in time (or space) over which GARD would process datasets.

Performing this split in time might be the most obvious (and goes along with splitting up the output files #11 )

Performing the split in space could be more efficient because GARD needs a complete time series for each grid point in the training period regardless of how much time is being processed. Would this end up writing separate files for each subset that need to be recombined later or could it write just the subset it processed to an output file that contains the entire grid?

Ideally both should be options, but do they need to be?

Tradeoffs in implementation complexity and benefit should be assessed briefly.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant