A maximum filesize can be specified as an argument to a `WalkBuilder`.
If a file exceeds the specified size it will be ignored as part of the
resulting file/directory set.
The filesize limit never applies to directories.
The documentation says:
> If `$XDG_CONFIG_HOME` is not set or is empty, then
> `$HOME/.config/git/ignore` is used instead.
This is the expected behavior, but the code looked at ~/git/ignore
instead.
When running ripgrep like this:
rg foo > output
we must be careful not to search `output` since ripgrep is actively writing
to it. Searching it can cause massive blowups where the file grows without
bound.
While this is conceptually easy to fix (check the inode of the redirection
and the inode of the file you're about to search), there are a few problems
with it.
First, inodes are a Unix thing, so we need a Windows specific solution to
this as well. To resolve this concern, I created a new crate, `same-file`,
which provides a cross platform abstraction.
Second, stat'ing every file is costly. This is not avoidable on Windows,
but on Unix, we can get the inode number directly from directory traversal.
However, this information wasn't exposed, but now it is (through both the
ignore and walkdir crates).
Fixes#286
This commit fixes two issues.
First, the iterator was executing the callback for every child of a
directory in a single thread. Therefore, if the walker was run over a
single directory, then no parallelism is used. We tweak the iterator
slightly so that we don't fall into this trap.
The second issue is a bit more subtle. In particular, we don't use the
blocking semantics of MsQueue because we *don't know when iteration
finishes*. This means that if there are a bunch of idle workers because
there is no work available to them, then they will spin and burn the
CPU. One case where this crops up is if you pipe the output of ripgrep
into `less` *and* the total number of files to search is fewer than the
number of threads ripgrep uses. We "fix" this with a very stupid
heuristic: when the queue yields no work, we sleep the thread for 1ms.
This still pegs the CPU, but not nearly as much as before.
If one really want to avoid this behavior when using ripgrep, then `-j1`
can be used to disable parallelism.
Fixes#258
When give an explicit file path on the command line like `foo` where `foo`
is a symlink, ripgrep should follow it even if `-L` isn't set. This is
consistent with the behavior of `foo/`.
Fixes#256
Previously, ignore::WalkParallel would invoke the callback for all
*explicitly* given file paths in a single thread, which effectively
meant that `rg pattern foo bar baz ...` didn't actually search foo, bar
and baz in parallel.
The code was structured that way to avoid spinning up workers if no
directory paths were given. The original intention was probably to have
a separate pool of threads responsible for searching, but ripgrep ended
up just reusing the ignore::WalkParallel workers themselves for searching,
and thereby subjected to its sub-par performance in this case.
The code has been restructured so that file paths are sent to the workers,
which brings back parallelism.
Fixes#226
Namely, passing a directory to --ignore-file caused ripgrep to allocate
memory without bound.
The issue was that I got a bit overzealous with partial error
reporting. Namely, when processing a gitignore file, we should try
to use every pattern even if some patterns are invalid globs (e.g.,
a**b). In the process, I applied the same logic to I/O errors. In this
case, it manifest by attempting to read lines from a directory, which
appears to yield Results forever, where each Result is an error of the
form "you can't read from a directory silly." Since I treated it as a
partial error, ripgrep was just spinning and accruing each error in
memory, which caused the OOM killer to kick in.
Fixes#228
This adds a new walk type in the `ignore` crate, `WalkParallel`, which
provides a way for recursively iterating over a set of paths in parallel
while respecting various ignore rules.
The API is a bit strange, as a closure producing a closure isn't
something one often sees, but it does seem to work well.
This also allowed us to simplify much of the worker logic in ripgrep
proper, where MultiWorker is now gone.
The name has_ignores is not descriptive in my opinion. I think
has_any_ignore_options more clearly states this method's purpose. I also
considered simply IgnoreOptions::any though I went with the more verbose
option.
