Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ripgrep not respecting global .gitignore #2392

Closed
Avishayy opened this issue Jan 6, 2023 · 6 comments
Closed

ripgrep not respecting global .gitignore #2392

Avishayy opened this issue Jan 6, 2023 · 6 comments
Labels
bug A bug. rollup A PR that has been merged with many others in a rollup.

Comments

@Avishayy
Copy link

Avishayy commented Jan 6, 2023

What version of ripgrep are you using?

ripgrep 13.0.0
-SIMD -AVX (compiled)
+SIMD +AVX (runtime)

How did you install ripgrep?

brew

What operating system are you using ripgrep on?

macOS Ventura 13.1

Describe your bug.

I have some local backup files in my dev repo and I'd like to ignore them from all rg functionalities. Since they exist on my local machine I don't add them to the repo tracked .gitignore, so I configured a global gitignore but it seems to not work. #9 seems to create support for it, but it's quite old already so perhaps it's now broken.

What are the steps to reproduce the behavior?

$ git config core.excludesFile
~/.gitignore

$ cat ~/.gitignore
*.sql

$ rg --files | rg backup
backup
backup_21_07_2022.sql
backup_11-12-22.sql
db_backup.sql

$ git add db_backup.sql
The following paths are ignored by one of your .gitignore files:
db_backup.sql
hint: Use -f if you really want to add them.
hint: Turn this message off by running
hint: "git config advice.addIgnoredFile false"

$ echo "*.sql" >> .gitignore

$ rg --files | rg backup
backup

You can see that git refuses to add the file since it is ignored by the global .gitignore and then that ripgrep would only respect it if is present in the working directory.

What is the actual behavior?

Patterns from global .gitignore are ignored by ripgrep

What is the expected behavior?

Patterns from global .gitignore are respected by ripgrep

What do you think ripgrep should have done?

Ignore the patterns present in the global .gitignore

@Avishayy
Copy link
Author

Avishayy commented Jan 6, 2023

Using the following paths for the global path doesn't work either:
$HOME/.gitignore
/Users/user/.gitignore
(but it does work for git as expected)
So it's not an issue of not resolving the tilde or something of sorts.

@BurntSushi
Copy link
Owner

Please include --debug output, as requested in the issue template.

Also, a small reproducible example that I can run would be helpful.

@Avishayy
Copy link
Author

Avishayy commented Jan 6, 2023

Sorry for not doing this yet, I found the root cause.
I edited the .gitconfig file manually and written the path as a string, i.e.

[core]
	excludesFile = "~/.gitignore"

(which git is happy with)

Changing it to a regular form

[core]
	excludesFile = ~/.gitignore

Makes ripgrep respect the global config as well, I'll attach a reproducer soon.

@Avishayy
Copy link
Author

Avishayy commented Jan 6, 2023

mkdir rg_global_gitignore
cd rg_global_gitignore
git init

# adding the config this way will strip the quotation marks
#git config --global core.excludesFile "~/.gitignore"

# open ~/.gitconfig and add:
#
# [core]
# 	excludesFile = "~/.gitignore"

echo "*.sql" > ~/.gitignore

touch backup backup.sql backup123.sql db_backup.sql
rg --files --debug

Output is

DEBUG|ignore::walk|crates/ignore/src/walk.rs:1741: ignoring ./.gitignore: Ignore(IgnoreMatch(Hidden))
DEBUG|ignore::walk|crates/ignore/src/walk.rs:1741: ignoring ./.git: Ignore(IgnoreMatch(Hidden))
backup.sql
backup
backup_21_07_2022.sql
backup123.sql
backup_11-12-22.sql
db_backup.sql

@BurntSushi
Copy link
Owner

My guess is that this can be fixed with a simple tweak to the regex used for parsing this file:

Regex::new(r"(?im)^\s*excludesfile\s*=\s*(.+)\s*$").unwrap();

@BurntSushi BurntSushi added the bug A bug. label Jan 6, 2023
zaidhaan added a commit to zaidhaan/ripgrep that referenced this issue Jan 12, 2023
Currently, ripgrep respects $HOME/.gitconfig and
$XDG_CONFIG_HOME/git/config, but not .git/config (the "local" or
"project" config). Git makes it such that .git/config overrides the
other two options, while still respecting the ./.gitignore. This commit
brings this functionality to ripgrep (while also cleaning a few
repetitive segments of code).

Fixes BurntSushi#2392
augustomelo added a commit to augustomelo/dotfiles that referenced this issue May 23, 2023
Decided to use a global git ignore so I could reuse on other tools (namely rg), for that to work rg looks for the
core.excludesFiles on the following files: `$HOME/.gitconfig` and `$XDG_CONFIG_HOME/git/config`.

Refernces:
BurntSushi/ripgrep#2396
BurntSushi/ripgrep#2392
@Kentokamoto
Copy link
Contributor

I'd like to take a look into fixing this since it looks like a pretty straight forward task. From what I was able to gather, this seems to be a problem when .gitconfig files have double quotes surrounding the value for excludesFile. If this is a small change to the regex under ripgrep/crates/ignore/src/gitignore.rs, then we probably might be able to get away with adding the double quote character to a character set along with whitespaces. So the new regex would look like

r"(?im)^\s*excludesfile\s*=[\x22\s]*(.+?)[\x22\s]*$"gm

@BurntSushi BurntSushi added the rollup A PR that has been merged with many others in a rollup. label Oct 15, 2023
BurntSushi pushed a commit that referenced this issue Oct 16, 2023
This permits the value to be surrounded in double quotes. It's still not
perfect, but probably better than it was. Getting this to be more
correct will likely require writing (or using) a real parser, which I'm
not particularly incliend to do at present.

Fixes #2392, Closes #2629
BurntSushi pushed a commit that referenced this issue Oct 16, 2023
This permits the value to be surrounded in double quotes. It's still not
perfect, but probably better than it was. Getting this to be more
correct will likely require writing (or using) a real parser, which I'm
not particularly incliend to do at present.

Fixes #2392, Closes #2629
netbsd-srcmastr pushed a commit to NetBSD/pkgsrc that referenced this issue Nov 28, 2023
14.0.2 (2023-11-27)
===================
This is a patch release with a few small bug fixes.

Bug fixes:

* [BUG #2654](BurntSushi/ripgrep#2654):
  Fix `deb` release sha256 sum file.
* [BUG #2658](BurntSushi/ripgrep#2658):
  Fix partial regression in the behavior of `--null-data --line-regexp`.
* [BUG #2659](BurntSushi/ripgrep#2659):
  Fix Fish shell completions.
* [BUG #2662](BurntSushi/ripgrep#2662):
  Fix typo in documentation for `-i/--ignore-case`.


14.0.1 (2023-11-26)
===================
This a patch release meant to fix `cargo install ripgrep` on Windows.

Bug fixes:

* [BUG #2653](BurntSushi/ripgrep#2653):
  Include `pkg/windows/Manifest.xml` in crate package.


14.0.0 (2023-11-26)
===================
ripgrep 14 is a new major version release of ripgrep that has some new
features, performance improvements and a lot of bug fixes.

The headlining feature in this release is hyperlink support. In this release,
they are an opt-in feature but may change to an opt-out feature in the future.
To enable them, try passing `--hyperlink-format default`. If you use [VS Code],
then try passing `--hyperlink-format vscode`. Please [report your experience
with hyperlinks][report-hyperlinks], positive or negative.

[VS Code]: https://code.visualstudio.com/
[report-hyperlinks]: BurntSushi/ripgrep#2611

Another headlining development in this release is that it contains a rewrite
of its regex engine. You generally shouldn't notice any changes, except for
some searches may get faster. You can read more about the [regex engine rewrite
on my blog][regex-internals]. Please [report your performance improvements or
regressions that you notice][report-perf].

[report-perf]: BurntSushi/ripgrep#2652

Finally, ripgrep switched the library it uses for argument parsing. Users
should not notice a difference in most cases (error messages have changed
somewhat), but flag overrides should generally be more consistent. For example,
things like `--no-ignore --ignore-vcs` work as one would expect (disables all
filtering related to ignore rules except for rules found in version control
systems such as `git`).

[regex-internals]: https://blog.burntsushi.net/regex-internals/

**BREAKING CHANGES**:

* `rg -C1 -A2` used to be equivalent to `rg -A2`, but now it is equivalent to
  `rg -B1 -A2`. That is, `-A` and `-B` no longer completely override `-C`.
  Instead, they only partially override `-C`.

Build process changes:

* ripgrep's shell completions and man page are now created by running ripgrep
with a new `--generate` flag. For example, `rg --generate man` will write a
man page in `roff` format on stdout. The release archives have not changed.
* The optional build dependency on `asciidoc` or `asciidoctor` has been
dropped. Previously, it was used to produce ripgrep's man page. ripgrep now
owns this process itself by writing `roff` directly.

Performance improvements:

* [PERF #1746](BurntSushi/ripgrep#1746):
  Make some cases with inner literals faster.
* [PERF #1760](BurntSushi/ripgrep#1760):
  Make most searches with `\b` look-arounds (among others) much faster.
* [PERF #2591](BurntSushi/ripgrep#2591):
  Parallel directory traversal now uses work stealing for faster searches.
* [PERF #2642](BurntSushi/ripgrep#2642):
  Parallel directory traversal has some contention reduced.

Feature enhancements:

* Added or improved file type filtering for Ada, DITA, Elixir, Fuchsia, Gentoo,
  Gradle, GraphQL, Markdown, Prolog, Raku, TypeScript, USD, V
* [FEATURE #665](BurntSushi/ripgrep#665):
  Add a new `--hyperlink-format` flag that turns file paths into hyperlinks.
* [FEATURE #1709](BurntSushi/ripgrep#1709):
  Improve documentation of ripgrep's behavior when stdout is a tty.
* [FEATURE #1737](BurntSushi/ripgrep#1737):
  Provide binaries for Apple silicon.
* [FEATURE #1790](BurntSushi/ripgrep#1790):
  Add new `--stop-on-nonmatch` flag.
* [FEATURE #1814](BurntSushi/ripgrep#1814):
  Flags are now categorized in `-h/--help` output and ripgrep's man page.
* [FEATURE #1838](BurntSushi/ripgrep#1838):
  An error is shown when searching for NUL bytes with binary detection enabled.
* [FEATURE #2195](BurntSushi/ripgrep#2195):
  When `extra-verbose` mode is enabled in zsh, show extra file type info.
* [FEATURE #2298](BurntSushi/ripgrep#2298):
  Add instructions for installing ripgrep using `cargo binstall`.
* [FEATURE #2409](BurntSushi/ripgrep#2409):
  Added installation instructions for `winget`.
* [FEATURE #2425](BurntSushi/ripgrep#2425):
  Shell completions (and man page) can be created via `rg --generate`.
* [FEATURE #2524](BurntSushi/ripgrep#2524):
  The `--debug` flag now indicates whether stdin or `./` is being searched.
* [FEATURE #2643](BurntSushi/ripgrep#2643):
  Make `-d` a short flag for `--max-depth`.
* [FEATURE #2645](BurntSushi/ripgrep#2645):
  The `--version` output will now also contain PCRE2 availability information.

Bug fixes:

* [BUG #884](BurntSushi/ripgrep#884):
  Don't error when `-v/--invert-match` is used multiple times.
* [BUG #1275](BurntSushi/ripgrep#1275):
  Fix bug with `\b` assertion in the regex engine.
* [BUG #1376](BurntSushi/ripgrep#1376):
  Using `--no-ignore --ignore-vcs` now works as one would expect.
* [BUG #1622](BurntSushi/ripgrep#1622):
  Add note about error messages to `-z/--search-zip` documentation.
* [BUG #1648](BurntSushi/ripgrep#1648):
  Fix bug where sometimes short flags with values, e.g., `-M 900`, would fail.
* [BUG #1701](BurntSushi/ripgrep#1701):
  Fix bug where some flags could not be repeated.
* [BUG #1757](BurntSushi/ripgrep#1757):
  Fix bug when searching a sub-directory didn't have ignores applied correctly.
* [BUG #1891](BurntSushi/ripgrep#1891):
  Fix bug when using `-w` with a regex that can match the empty string.
* [BUG #1911](BurntSushi/ripgrep#1911):
  Disable mmap searching in all non-64-bit environments.
* [BUG #1966](BurntSushi/ripgrep#1966):
  Fix bug where ripgrep can panic when printing to stderr.
* [BUG #2046](BurntSushi/ripgrep#2046):
  Clarify that `--pre` can accept any kind of path in the documentation.
* [BUG #2108](BurntSushi/ripgrep#2108):
  Improve docs for `-r/--replace` syntax.
* [BUG #2198](BurntSushi/ripgrep#2198):
  Fix bug where `--no-ignore-dot` would not ignore `.rgignore`.
* [BUG #2201](BurntSushi/ripgrep#2201):
  Improve docs for `-r/--replace` flag.
* [BUG #2288](BurntSushi/ripgrep#2288):
  `-A` and `-B` now only each partially override `-C`.
* [BUG #2236](BurntSushi/ripgrep#2236):
  Fix gitignore parsing bug where a trailing `\/` resulted in an error.
* [BUG #2243](BurntSushi/ripgrep#2243):
  Fix `--sort` flag for values other than `path`.
* [BUG #2246](BurntSushi/ripgrep#2246):
  Add note in `--debug` logs when binary files are ignored.
* [BUG #2337](BurntSushi/ripgrep#2337):
  Improve docs to mention that `--stats` is always implied by `--json`.
* [BUG #2381](BurntSushi/ripgrep#2381):
  Make `-p/--pretty` override flags like `--no-line-number`.
* [BUG #2392](BurntSushi/ripgrep#2392):
  Improve global git config parsing of the `excludesFile` field.
* [BUG #2418](BurntSushi/ripgrep#2418):
  Clarify sorting semantics of `--sort=path`.
* [BUG #2458](BurntSushi/ripgrep#2458):
  Make `--trim` run before `-M/--max-columns` takes effect.
* [BUG #2479](BurntSushi/ripgrep#2479):
  Add documentation about `.ignore`/`.rgignore` files in parent directories.
* [BUG #2480](BurntSushi/ripgrep#2480):
  Fix bug when using inline regex flags with `-e/--regexp`.
* [BUG #2505](BurntSushi/ripgrep#2505):
  Improve docs for `--vimgrep` by mentioning footguns and some work-arounds.
* [BUG #2519](BurntSushi/ripgrep#2519):
  Fix incorrect default value in documentation for `--field-match-separator`.
* [BUG #2523](BurntSushi/ripgrep#2523):
  Make executable searching take `.com` into account on Windows.
* [BUG #2574](BurntSushi/ripgrep#2574):
  Fix bug in `-w/--word-regexp` that would result in incorrect match offsets.
* [BUG #2623](BurntSushi/ripgrep#2623):
  Fix a number of bugs with the `-w/--word-regexp` flag.
* [BUG #2636](BurntSushi/ripgrep#2636):
  Strip release binaries for macOS.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug A bug. rollup A PR that has been merged with many others in a rollup.
Projects
None yet
Development

Successfully merging a pull request may close this issue.

4 participants
@Kentokamoto @BurntSushi @Avishayy and others