Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Out of memory listing entries on one human chromosome on a machine with 300 GB ram and 165 GB BigBed file #148

Open
balwierz opened this issue Apr 19, 2024 · 1 comment

Comments

@balwierz
Copy link

I thought I could use this library for querying extremely large BigWig files, but despite providing python with 300 GB of ram it crashes anyway.
Is pyBigWig keeping all the data in memory at any point of operation?

@balwierz balwierz changed the title pyBigWig.open() runs out of memory on 300 GB ram and 165 GB BigBed file Out of memory listing entries on one human chromosome on a machine with 300 GB ram and 165 GB BigBed file Apr 22, 2024
@dpryan79
Copy link
Collaborator

It'll need to keep those results in memory to output them. I wonder what the utility of such a large bigBed file is, since I suspect you'd be better served by storing some sort of summarization of the underlying data.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants