Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Slow lazy loading in BlackrockIO #392

Open
JuliaSprenger opened this issue Sep 21, 2017 · 2 comments
Open

Slow lazy loading in BlackrockIO #392

JuliaSprenger opened this issue Sep 21, 2017 · 2 comments

Comments

@JuliaSprenger
Copy link
Member

Currently lazy loading is implemented by loading the complete dataset in order to determine the size of data objects, in particular AnalogSignals. Although it is faster than loading without lazy loading it could be very much improved by calculating the size based on the binning of the data and the provided t_start and t_stop values.

@samuelgarcia
Copy link
Contributor

In the new rawio and BaseFromRaw, this is done done as follow:

  • lazy=False, time_slice=None: the data is read entirely
  • lazy=True, time_slice=None: the data is NOT read but lazy_shape give the full shape of Analogsignals
  • lazy=False, time_slice=(t1, t2): the data is read only between t1 and t2
  • lazy=True , time_slice=(t1, t2): is not allowed, it raise error, this is a non sens to ask for a shape in range you knwon in before.

@JuliaSprenger
Copy link
Member Author

Actually considering that t1 and t2 might need to be adjusted during the loading process (eg. as done in the NeuralynxIO due to gaps in the data) also the last case makes sense and might be useful.

@apdavison apdavison added this to the future milestone Sep 26, 2017
@apdavison apdavison added IO and removed IO labels Jan 9, 2018
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

3 participants