This also replaces '--all' in Cargo commands with '--workspace'. The
former has apparently been deprecated.
We also fix a couple warnings that this new step detected.
Closes#1848
If ripgrep was called in a way where the entire contents of a file
aren't read (like --files-with-matches, among other methods), and if the
file was read through an external process, then ripgrep would never reap
that process.
We fix this by introducing an explicit 'close' method, which we now call
when using decompression or preprocessor searches.
The implementation of 'close' is a little hokey. In particular, when we
close stdout, this usually results in a broken pipe, and, consequently,
a non-zero code returned once the child process is reaped. This is
"situation normal," so we invent a (hopefully portable) heuristic for
detecting it.
Fixes#1766, Closes#1767
This fixes a bug only present on Windows that would permit someone to
execute an arbitrary program if they crafted an appropriate directory
tree. Namely, if someone put an executable named 'xz.exe' in the root of
a directory tree and one ran 'rg -z foo' from the root of that tree,
then the 'xz.exe' executable in that tree would execute if there are any
'xz' files anywhere in the tree.
The root cause of this problem is that 'CreateProcess' on Windows will
implicitly look in the current working directory for an executable when
it is given a relative path to a program. Rust's standard library allows
this behavior to occur, so we work around it here. We work around it by
explicitly resolving programs like 'xz' via 'PATH'. That way, we only
ever pass an absolute path to 'CreateProcess', which avoids the implicit
behavior of checking the current working directory.
This fix doesn't apply to non-Windows systems as it is believed to only
impact Windows. In theory, the bug could apply on Unix if '.' is in
one's PATH, but at that point, you reap what you sow.
While the extent to which this is a security problem isn't clear, I
think users generally expect to be able to download or clone
repositories from the Internet and run ripgrep on them without fear of
anything too awful happening. Being able to execute an arbitrary program
probably violates that expectation. Therefore, CVE-2021-3013[1] was
created for this issue.
We apply the same logic to the --pre command, since the --pre command is
likely in a user's config file and it would be surprising for something
that the user is searching to modify which preprocessor command is used.
The --pre and -z/--search-zip flags are the only two ways that ripgrep
will invoke external programs, so this should cover any possible
exploitable cases of this bug.
[1] - https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2021-3013
This fixes a bug where using \A or (?-m)^ in combination with
-U/--multiline would permit matches that aren't anchored to the
beginning of the file. The underlying cause was an optimization that
occurred when mmaps couldn't be used. Namely, ripgrep tries to still
read the input incrementally if it knows the pattern can't match through
a new line. But the detection logic was flawed, since it didn't account
for line anchors. This commit fixes that.
Fixes#1878, Fixes#1879
It turned out that --vimgrep wasn't quite getting the column of each
match correctly. Instead of printing column numbers relative to the
current line, it was printing column numbers as byte offsets relative to
where the match began. To fix this, we simply subtract the offset of the
line number from the beginning of the match. If the beginning of the
match came before the start of the current line, then there's really
nothing sensible we can do other than to use a column number of 1, which
we now document.
Interestingly, existing tests were checking that the previous behavior
was intended. My only defense is that I somehow tricked myself into
thinking it was a byte offset instead of a column number.
Kudos to @bfrg for calling this out in #1866:
https://github.com/BurntSushi/ripgrep/issues/1866#issuecomment-841635553
It looks like no new dependencies have been introduced. Yay!
This update was primarily motivated to bring regex 1.5 in with its new
memmem implementation from the memchr crate.
This message will emit the binary detection mechanism being used for
each file.
This does not noticeably increases the number of log messages, as the
'trace' level is already used for emitting messages for every file
searched.
This trace message was added in the course of investigating #1838.
Sadly, there were several tests that are coupled to the size of the
buffer used by ripgrep. Making the tests agnostic to the size is
difficult. And it's annoying to fix the tests. But we rarely change the
buffer size, so ¯\_(ツ)_/¯.
This increases the initial buffer size from 8KB to 64KB. This actually
leads to a reasonably noticeable improvement in at least one work-load,
and is unlikely to regress in any other case. Also, since Rust programs
(at least on Linux) seem to always use a minimum of 6-8MB of memory,
adding an extra 56KB is negligible.
Before:
$ hyperfine -i "rg 'zqzqzqzq' OpenSubtitles2018.raw.en --no-mmap"
Benchmark #1: rg 'zqzqzqzq' OpenSubtitles2018.raw.en --no-mmap
Time (mean ± σ): 2.109 s ± 0.012 s [User: 565.5 ms, System: 1541.6 ms]
Range (min … max): 2.094 s … 2.128 s 10 runs
After:
$ hyperfine -i "rg 'zqzqzqzq' OpenSubtitles2018.raw.en --no-mmap"
Benchmark #1: rg 'zqzqzqzq' OpenSubtitles2018.raw.en --no-mmap
Time (mean ± σ): 1.802 s ± 0.006 s [User: 462.3 ms, System: 1337.9 ms]
Range (min … max): 1.795 s … 1.814 s 10 runs
If ripgrep hasn't been built yet (i.e. target/debug/ doesn't exist),
then cargo-out-dir can't find OUT_DIR and the copy commands fail. Fix by
running cargo build before finding OUT_DIR.
Also add a check to fail early with a sensible error message when
asciidoctor isn't installed, rather than failing because of a missing
rg.1 file after the build.
PR #1831
memmap is unmaintained at this point and it is being flagged as a
RUSTSEC advisory in ripgrep. This doesn't seem like that big of a deal
to me honestly, but memmap2 looks like a fine choice at this point.
Fixes#1785, Closes#1786
It seems that PowerShell uses sockets instead of FIFOs to redirect the
output between commands. So add `is_socket` to our `is_readable_stdin`
check.
This seems unlikely to cause problems and it probably more generally
correct than what we had before. In theory, it could cause problems if
it produces false positives, in which case, ripgrep will try to read
stdin when it should search the current working directory. (And this
usually winds up manifesting as ripgrep blocking forever.) But, if the
stdin handle reports itself as a socket, then it seems like we should
read it.
Fixes#1741, Closes#1742
`ignore::Error` wraps `std::io::Error` with additional information
(as well as expose non-IO errors). For people wanting to inspect what
the error is, they have to recursively match the Enum. This provides
`io_error` and `into_io_error` helpers to do this for the user.
PR #1740
This comes up as a corner case where folks provide -e/--regexp in a
configuration file and then expect to be able to run 'rg' with no args.
However, ripgrep fails because it still expects at least one pattern
even though one was specified in the config file.
This occurs because ripgrep has to parse its CLI parameters before
reading the config file. (For log output settings and to handle the
--no-config flag.) This initial parse will fail if there are no patterns
specified.
The only way to solve this that I can see is to somehow relax the
requirements of the initial parse. But this is problematic because we
would still need to enforce those requirements in cases where we don't
do a second parse (when no config file is present).
All in all, this doesn't seem like a problem that is worth solving.
Closes#1730
This brings in all other semver updates.
This did require updating some tests, since bstr changed its debug
output for NUL bytes to be a bit more idiomatic.
This updates encoding_rs, crossbeam-utils and crossbeam-channel. This
serves two purposes. The encoding_rs update fixes a compilation failure
on the latest nightly. The crossbeam updates are good sense and to
reduce duplicate dependencies such as cfg-if. (Although, we note that
the log crate still pulls in cfg-if 0.1, so ripgrep has a duplicate
dependency there for now. But it's very small.)
Fixes#1721, Closes#1705
Bazel supports `BUILD.bazel` as well as `WORKSPACE.bazel`. In
addition, it is common to ship BUILD/WORKSPACE templates for
external repositories suffixed with .bazel for easier tool
recognition.
Co-authored-by: Brandon Adams <brandon.adams@imc.com>
PR #1716
