Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Retroactively shard dataset into subdatasets #2978

Open
effigies opened this issue Jan 23, 2024 · 2 comments
Open

Retroactively shard dataset into subdatasets #2978

effigies opened this issue Jan 23, 2024 · 2 comments
Labels

Comments

@effigies
Copy link
Contributor

What would you like to see added?

If a dataset becomes too large, it would be good to turn subject directories into submodules. For tagging purposes, we would need to find some way of identifying the old and new commits (grafting?).

At least for datasets with new subjects, one possibility would be to shard new subjects, but new sessions within existing subjects may or may not need to be handled differently.

Derivatives and sourcedata would also be candidates for sharding.

Alternatives

No response

Do you have any interest in helping implement the feature?

No

Additional information / screenshots

No response

@effigies
Copy link
Contributor Author

Related: datalad/datalad#3554

@yarikoptic
Copy link
Contributor

do you have examples of already existing openneuro datasets and how would you like to split?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

2 participants