Specific points index

[martindurant]

Instead of creating seek points every x bytes, is it theoretically possible to pick specific points in the data stream that we know we will be needing? If yes, is there any appetite for implementing this?

[pauldmccarthy]

Hi @martindurant, for gzip/deflate streams I don't think it would be possible to create index points at exact locations, as the zran functions calls the zlib inflate function so as to stop inflation at deflate block boundaries, which are somewhat arbitrarily placed in a typical gzip file (although things would be different if you controlled creation of the file).

However, it would probably not be too much work to adjust the code to create index points at approximately the desired locations - the logic controlling index point creation (excluding points at the start/end of the stream) is here. I suppose the set of desired locations could be passed in at creation and checked here, and a new bit flag which controls whether to create points at spacing intervals, or at the specified locations..

[martindurant]

Understood, thanks. The use case here would be individual files within a tar.gz archive, or contiguous array buffers within any of several file formats that store data that way. Those all would have exact byte start/stop values, of course, which won't align with gzip boundaries - but if you can get the best previous boundary for each location, then seek and read will work pretty well.