-
-
Notifications
You must be signed in to change notification settings - Fork 2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Memory not freed in ignore crate when ignore flags are enabled #2690
Labels
bug
A bug.
Comments
BurntSushi
pushed a commit
to fe9lix/ripgrep
that referenced
this issue
Jan 6, 2024
It looks like there is a reference cycle caused by the compiled matchers (compiled HashMap holds ref to Ignore and Ignore holds ref to HashMap). Using weak refs fixes issue BurntSushi#2690 in my test project. Also confirmed via before and after when profiling the code, see the attached screenshots in BurntSushi#2692. Fixes BurntSushi#2690
netbsd-srcmastr
pushed a commit
to NetBSD/pkgsrc
that referenced
this issue
Jan 8, 2024
14.1.0 ====== This is a minor release with a few small new features and bug fixes. This release contains a bug fix for unbounded memory growth while walking a directory tree. This release also includes improvements to the completions for the `fish` shell, and release binaries for several additional ARM targets. Bug fixes: * [BUG #2664](BurntSushi/ripgrep#2690): Fix unbounded memory growth in the `ignore` crate. Feature enhancements: * Added or improved file type filtering for Lean and Meson. * [FEATURE #2684](BurntSushi/ripgrep#2684): Improve completions for the `fish` shell. * [FEATURE #2702](BurntSushi/ripgrep#2702): Add release binaries for `armv7-unknown-linux-gnueabihf`, `armv7-unknown-linux-musleabihf` and `armv7-unknown-linux-musleabi`.
This comment was marked as spam.
This comment was marked as spam.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Please tick this box to confirm you have reviewed the above.
What version of ripgrep are you using?
ignore crate 0.4.21
How did you install ripgrep?
cargo add ignore
What operating system are you using ripgrep on?
macOS Sonoma 14.1.1
Describe your bug.
ignore's WalkParallel does not seem to free memory when ignore flags are enabled.
What are the steps to reproduce the behavior?
Run the "stress test" example project (adjust the path and timeout) and observe the memory growth over time in some process monitoring tool. I'm compiling for the ARM aarch64-apple-darwin target and running it on an M1 Macbook Pro.
Run heap profiling tools to confirm the leaks. I used Instruments' Allocations and Leaks via
cargo instruments
on macOS (latest stable Rust toolchain) and LeakSanitizer via the the nightly toolchain and compiled for x86 (RUSTFLAGS="-Z sanitizer=leak" cargo run --target x86_64-apple-darwin
). Visual output with stack traces is appended for both the leaking and non-leaking versions (ignore flags disabled), the text output of LeakSanitizer is attached, and the visual and text output of heaptrack (linux VM only).ignore-mem.zip
What is the actual behavior?
Memory usage increases over time, unbounded – but only when the ignore flags are enabled. The memory growth increases faster when custom overrides are enabled. There is no continuous memory growth when the ignore flags are disabled (regardless of whether custom overrides are enabled or not).
What is the expected behavior?
I would expect that memory usage would be bounded and not grow over time when the threads have been joined and the WalkParallel instance is dropped in the example project. Memory would stay around a baseline level as the allocator reuses freed memory. This happens when the ignore flags are disabled. To exclude memory fragmentation as a possible cause, I've also tested this with jemalloc and observed the same behavior (with a different baseline memory usage and allocation pattern but still unbounded growth).
Since I've also tested older version of this crate and observed the same behavior, it makes me think that I'm 1) missing something obvious, or 2) misusing the crate and its API, 3) holding misconceptions about how memory is supposed to be managed in the example. If so, I'd be happy to be pointed in the right direction how to correctly use the crate for a long-running process that walks different subdirectory trees at certain points in time, while keeping memory bounded within a baseline range.
The text was updated successfully, but these errors were encountered: