Previously, ignore::WalkParallel would invoke the callback for all
*explicitly* given file paths in a single thread, which effectively
meant that `rg pattern foo bar baz ...` didn't actually search foo, bar
and baz in parallel.
The code was structured that way to avoid spinning up workers if no
directory paths were given. The original intention was probably to have
a separate pool of threads responsible for searching, but ripgrep ended
up just reusing the ignore::WalkParallel workers themselves for searching,
and thereby subjected to its sub-par performance in this case.
The code has been restructured so that file paths are sent to the workers,
which brings back parallelism.
Fixes#226
Namely, passing a directory to --ignore-file caused ripgrep to allocate
memory without bound.
The issue was that I got a bit overzealous with partial error
reporting. Namely, when processing a gitignore file, we should try
to use every pattern even if some patterns are invalid globs (e.g.,
a**b). In the process, I applied the same logic to I/O errors. In this
case, it manifest by attempting to read lines from a directory, which
appears to yield Results forever, where each Result is an error of the
form "you can't read from a directory silly." Since I treated it as a
partial error, ripgrep was just spinning and accruing each error in
memory, which caused the OOM killer to kick in.
Fixes#228
This flag is similar to what's found in grep: it will suppress all error
messages, such as those shown when a particular file couldn't be read.
Closes#149
This was a subtle bug, but the big picture was that the smart case
information wasn't being carried through to the literal extraction in
some cases. When this happened, it was possible to get back an incomplete
set of literals, which would therefore miss some valid matches.
The fix to this is to actually parse the regex and determine whether
smart case applies before doing anything else. It's a little extra work,
but parsing is pretty fast.
Fixes#199
This adds a new walk type in the `ignore` crate, `WalkParallel`, which
provides a way for recursively iterating over a set of paths in parallel
while respecting various ignore rules.
The API is a bit strange, as a closure producing a closure isn't
something one often sees, but it does seem to work well.
This also allowed us to simplify much of the worker logic in ripgrep
proper, where MultiWorker is now gone.
The name has_ignores is not descriptive in my opinion. I think
has_any_ignore_options more clearly states this method's purpose. I also
considered simply IgnoreOptions::any though I went with the more verbose
option.
