.skm file format on disk

[johnlees]

For 2M samples this is about 100Mb. If it gets really large, implementing an on-disk key-value store (redis, rocksDB or just mysql) may be necessary.

[johnlees]

(NB: arrow is in-memory so not useful here)

SQL would be good for a fixed object type and allowing queries. Maybe adding basic metadata fields (species, time, location) could be a nice addition.

Key-value stores are unstructured so allow changes in future (SQL also works for this, with a JSON as the value). To allow queries on this you'd need to use elastic or similar.

This starts to feel like the job of an external tool though as we really just need the index to load here (and names can be consistent). So maybe key-value is the right choice.