-
Notifications
You must be signed in to change notification settings - Fork 0
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
bundleSize is not a configurable WriterOption #2
Comments
I believe I also have a case where I would want bundle size to be configurable. I am writing out many rows per file to many files, and when I hit ~800 open FileWriters my process OOMs, even with a 20GB heap size. AFAICT the only collection that is growing is the map of FileWriters. |
The batchQueue is a BlockingQueue that can't hold more than 10 bundles and will start blocking beyond that. There is no particular reason for this value and we could probably lower that to 1 bundle. This would lower the memory usage in the case where the calling thread outruns the dendrite writing thread and shouldn't have any noticeable write throughput impact. I would rather have a good hardcoded value than expose more internals in the public API. Would that address both of your needs? |
Made the batch queue size change here 7ab0292 |
Fair enough. This covers my issue. If @harob agrees, we can close this out. |
Currently, bundle size is defaulted to 256: https://github.com/jwhitbeck/dendrite/blob/master/dendrite/java-src/dendrite/java/Options.java#L343. If the write thread is outrun by the writing thread, it's possible that the batchQueue gets filled. This is a problem if the machine running dendrite doesn't have enough memory.
Another method to solve this problem might be to apply backpressure via the API (i.e. making certain write calls more expensive when a flush is necessary).
The text was updated successfully, but these errors were encountered: