-
Notifications
You must be signed in to change notification settings - Fork 1
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
.skm file format on disk #17
Comments
(NB: arrow is in-memory so not useful here) SQL would be good for a fixed object type and allowing queries. Maybe adding basic metadata fields (species, time, location) could be a nice addition. Key-value stores are unstructured so allow changes in future (SQL also works for this, with a JSON as the value). To allow queries on this you'd need to use elastic or similar. This starts to feel like the job of an external tool though as we really just need the index to load here (and names can be consistent). So maybe key-value is the right choice. |
For 2M samples this is about 100Mb. If it gets really large, implementing an on-disk key-value store (redis, rocksDB or just mysql) may be necessary.
The text was updated successfully, but these errors were encountered: