1
0
mirror of https://github.com/BurntSushi/ripgrep.git synced 2024-12-12 19:18:24 +02:00
ripgrep recursively searches directories for a regex pattern while respecting your gitignore
Go to file
Andrew Gallant e5bb750995
ignore: add 'stdout' skipping to the walker
This commit adds a new 'skip_stdout' option to the directory walker. When
enabled, it will skip yielding any directory entries that are believed to
correspond to stdout for the current process. This is useful for filtering
out 'results' in a command like 'grep -r foo > results' in order to avoid
an unbounded feedback mechanism.
2018-08-27 21:18:53 -04:00
benchsuite benchsuite: fix formatting 2018-01-08 19:23:43 -05:00
ci release: better support for binary Debian package 2018-08-21 23:05:52 -04:00
complete complete: don't complete bare pattern after -f 2018-08-27 07:56:40 -04:00
doc doc: update ripgrep's description 2018-08-21 23:05:52 -04:00
globset ripgrep: migrate to libripgrep 2018-08-20 07:10:19 -04:00
grep deps: update walkdir minimum version 2018-08-23 09:54:45 -04:00
grep-matcher deps: update libripgrep crate versions 2018-08-20 17:34:45 -04:00
grep-pcre2 deps: update libripgrep crate versions 2018-08-20 17:34:45 -04:00
grep-printer grep-printer: fix bug in printing truncated lines 2018-08-21 23:05:52 -04:00
grep-regex deps: update libripgrep crate versions 2018-08-20 17:34:45 -04:00
grep-searcher deps: update libripgrep crate versions 2018-08-20 17:34:45 -04:00
ignore ignore: add 'stdout' skipping to the walker 2018-08-27 21:18:53 -04:00
pkg/brew pkg: update brew tap to 0.9.0 2018-08-03 17:04:36 -04:00
src ripgrep: add --sort and --sortr flags 2018-08-26 18:42:25 -04:00
termcolor termcolor: moved to its own repository 2018-07-17 18:03:38 -04:00
tests tests: touch up tests on Windows 2018-08-21 23:05:52 -04:00
wincolor termcolor: moved to its own repository 2018-07-17 18:03:38 -04:00
.gitignore ci: update deployment for doc rearrangement 2018-02-10 12:12:47 -05:00
.travis.yml ripgrep: move minimum version to Rust stable 2018-08-21 23:05:52 -04:00
appveyor.yml ripgrep: migrate to libripgrep 2018-08-20 07:10:19 -04:00
build.rs ripgrep: move minimum version to Rust stable 2018-08-21 23:05:52 -04:00
Cargo.lock ignore: add 'same_file_system' option 2018-08-26 18:42:25 -04:00
Cargo.toml ripgrep: use winapi-util for stdin_is_readable 2018-08-25 00:30:15 -04:00
CHANGELOG.md changelog: fix typo 2018-08-23 13:17:27 -04:00
COPYING
FAQ.md doc: fix typo 2018-08-20 18:00:14 -04:00
GUIDE.md doc: sync config file examples 2018-05-25 06:42:05 -04:00
HomebrewFormula
ISSUE_TEMPLATE.md doc: go away snap 2018-04-30 15:25:51 -04:00
LICENSE-MIT
README.md doc: clarify use of SIMD features 2018-08-23 09:56:37 -04:00
snapcraft.yaml snap: build without wrappers 2018-07-21 13:28:27 -04:00
UNLICENSE

ripgrep (rg)

ripgrep is a line-oriented search tool that recursively searches your current directory for a regex pattern while respecting your gitignore rules. ripgrep has first class support on Windows, macOS and Linux, with binary downloads available for every release. ripgrep is similar to other popular search tools like The Silver Searcher, ack and grep.

Linux build status Windows build status Crates.io

Dual-licensed under MIT or the UNLICENSE.

CHANGELOG

Please see the CHANGELOG for a release history.

Screenshot of search results

A screenshot of a sample search with ripgrep

Quick examples comparing tools

This example searches the entire Linux kernel source tree (after running make defconfig && make -j8) for [A-Z]+_SUSPEND, where all matches must be words. Timings were collected on a system with an Intel i7-6900K 3.2 GHz, and ripgrep was compiled with SIMD enabled.

Please remember that a single benchmark is never enough! See my blog post on ripgrep for a very detailed comparison with more benchmarks and analysis.

Tool Command Line count Time
ripgrep (Unicode) rg -n -w '[A-Z]+_SUSPEND' 450 0.106s
git grep LC_ALL=C git grep -E -n -w '[A-Z]+_SUSPEND' 450 0.553s
The Silver Searcher ag -w '[A-Z]+_SUSPEND' 450 0.589s
git grep (Unicode) LC_ALL=en_US.UTF-8 git grep -E -n -w '[A-Z]+_SUSPEND' 450 2.266s
sift sift --git -n -w '[A-Z]+_SUSPEND' 450 3.505s
ack ack -w '[A-Z]+_SUSPEND' 1878 6.823s
The Platinum Searcher pt -w -e '[A-Z]+_SUSPEND' 450 14.208s

(Yes, ack has a bug.)

Here's another benchmark that disregards gitignore files and searches with a whitelist instead. The corpus is the same as in the previous benchmark, and the flags passed to each command ensure that they are doing equivalent work:

Tool Command Line count Time
ripgrep rg -L -u -tc -n -w '[A-Z]+_SUSPEND' 404 0.079s
ucg ucg --type=cc -w '[A-Z]+_SUSPEND' 390 0.163s
GNU grep egrep -R -n --include='*.c' --include='*.h' -w '[A-Z]+_SUSPEND' 404 0.611s

(ucg has slightly different behavior in the presence of symbolic links.)

And finally, a straight-up comparison between ripgrep and GNU grep on a single large file (~9.3GB, OpenSubtitles2016.raw.en.gz):

Tool Command Line count Time
ripgrep rg -w 'Sherlock [A-Z]\w+' 5268 2.108s
GNU grep LC_ALL=C egrep -w 'Sherlock [A-Z]\w+' 5268 7.014s

In the above benchmark, passing the -n flag (for showing line numbers) increases the times to 2.640s for ripgrep and 10.277s for GNU grep.

Why should I use ripgrep?

  • It can replace many use cases served by other search tools because it contains most of their features and is generally faster. (See the FAQ for more details on whether ripgrep can truly replace grep.)
  • Like other tools specialized to code search, ripgrep defaults to recursive directory search and won't search files ignored by your .gitignore files. It also ignores hidden and binary files by default. ripgrep also implements full support for .gitignore, whereas there are many bugs related to that functionality in other code search tools claiming to provide the same functionality.
  • ripgrep can search specific types of files. For example, rg -tpy foo limits your search to Python files and rg -Tjs foo excludes Javascript files from your search. ripgrep can be taught about new file types with custom matching rules.
  • ripgrep supports many features found in grep, such as showing the context of search results, searching multiple patterns, highlighting matches with color and full Unicode support. Unlike GNU grep, ripgrep stays fast while supporting Unicode (which is always on).
  • ripgrep supports searching files in text encodings other than UTF-8, such as UTF-16, latin-1, GBK, EUC-JP, Shift_JIS and more. (Some support for automatically detecting UTF-16 is provided. Other text encodings must be specifically specified with the -E/--encoding flag.)
  • ripgrep supports searching files compressed in a common format (gzip, xz, lzma, bzip2 or lz4) with the -z/--search-zip flag.
  • ripgrep supports arbitrary input preprocessing filters which could be PDF text extraction, less supported decompression, decrypting, automatic encoding detection and so on.

In other words, use ripgrep if you like speed, filtering by default, fewer bugs, and Unicode support.

Why shouldn't I use ripgrep?

Despite initially not wanting to add every feature under the sun to ripgrep, over time, ripgrep has grown support for most features found in other file searching tools. This includes searching for results spanning across multiple lines, and opt-in support for PCRE2, which provides look-around and backreference support.

At this point, the primary reasons not to use ripgrep probably consist of one or more of the following:

  • You need a portable and ubiquitous tool. While ripgrep works on Windows, macOS and Linux, it is not ubiquitous and it does not conform to any standard such as POSIX. The best tool for this job is good old grep.
  • There still exists some other minor feature (or bug) found in another tool that isn't in ripgrep.
  • There is a performance edge case where ripgrep doesn't do well where another tool does do well. (Please file a bug report!)
  • ripgrep isn't possible to install on your machine or isn't available for your platform. (Please file a bug report!)

Is it really faster than everything else?

Generally, yes. A large number of benchmarks with detailed analysis for each is available on my blog.

Summarizing, ripgrep is fast because:

  • It is built on top of Rust's regex engine. Rust's regex engine uses finite automata, SIMD and aggressive literal optimizations to make searching very fast. (PCRE2 support can be opted into with the -P/--pcre2 flag.)
  • Rust's regex library maintains performance with full Unicode support by building UTF-8 decoding directly into its deterministic finite automaton engine.
  • It supports searching with either memory maps or by searching incrementally with an intermediate buffer. The former is better for single files and the latter is better for large directories. ripgrep chooses the best searching strategy for you automatically.
  • Applies your ignore patterns in .gitignore files using a RegexSet. That means a single file path can be matched against multiple glob patterns simultaneously.
  • It uses a lock-free parallel recursive directory iterator, courtesy of crossbeam and ignore.

Feature comparison

Andy Lester, author of ack, has published an excellent table comparing the features of ack, ag, git-grep, GNU grep and ripgrep: https://beyondgrep.com/feature-comparison/

Note that ripgrep has grown a few significant new features recently that are not yet present in Andy's table. This includes, but is not limited to, configuration files, passthru, support for searching compressed files, multiline search and opt-in fancy regex support via PCRE2.

Installation

The binary name for ripgrep is rg.

Archives of precompiled binaries for ripgrep are available for Windows, macOS and Linux. Users of platforms not explicitly mentioned below are advised to download one of these archives.

Linux binaries are static executables. Windows binaries are available either as built with MinGW (GNU) or with Microsoft Visual C++ (MSVC). When possible, prefer MSVC over GNU, but you'll need to have the Microsoft VC++ 2015 redistributable installed.

If you're a macOS Homebrew or a Linuxbrew user, then you can install ripgrep either from homebrew-core, (compiled with rust stable, no SIMD):

$ brew install ripgrep

or you can install a binary compiled with rust nightly (including SIMD and all optimizations) by utilizing a custom tap:

$ brew tap burntsushi/ripgrep https://github.com/BurntSushi/ripgrep.git
$ brew install ripgrep-bin

If you're a MacPorts user, then you can install ripgrep from the official ports:

$ sudo port install ripgrep

If you're a Windows Chocolatey user, then you can install ripgrep from the official repo:

$ choco install ripgrep

If you're a Windows Scoop user, then you can install ripgrep from the official bucket:

$ scoop install ripgrep

If you're an Arch Linux user, then you can install ripgrep from the official repos:

$ pacman -S ripgrep

If you're a Gentoo user, you can install ripgrep from the official repo:

$ emerge sys-apps/ripgrep

If you're a Fedora 27+ user, you can install ripgrep from official repositories.

$ sudo dnf install ripgrep

If you're a Fedora 24+ user, you can install ripgrep from copr:

$ sudo dnf copr enable carlwgeorge/ripgrep
$ sudo dnf install ripgrep

If you're an openSUSE Tumbleweed user, you can install ripgrep from the official repo:

$ sudo zypper install ripgrep

If you're a RHEL/CentOS 7 user, you can install ripgrep from copr:

$ sudo yum-config-manager --add-repo=https://copr.fedorainfracloud.org/coprs/carlwgeorge/ripgrep/repo/epel-7/carlwgeorge-ripgrep-epel-7.repo
$ sudo yum install ripgrep

If you're a Nix user, you can install ripgrep from nixpkgs:

$ nix-env --install ripgrep
$ # (Or using the attribute name, which is also ripgrep.)

If you're a Debian user (or a user of a Debian derivative like Ubuntu), then ripgrep can be installed using a binary .deb file provided in each ripgrep release. Note that ripgrep is not in the official Debian or Ubuntu repositories.

$ curl -LO https://github.com/BurntSushi/ripgrep/releases/download/0.9.0/ripgrep_0.9.0_amd64.deb
$ sudo dpkg -i ripgrep_0.9.0_amd64.deb

If you run Debian Buster (currently Debian testing) or Debian sid, ripgrep is officially maintained by Debian.

$ sudo apt-get install ripgrep

(N.B. Various snaps for ripgrep on Ubuntu are also available, but none of them seem to work right and generate a number of very strange bug reports that I don't know how to fix and don't have the time to fix. Therefore, it is no longer a recommended installation option.)

If you're a FreeBSD user, then you can install ripgrep from the official ports:

# pkg install ripgrep

If you're an OpenBSD user, then you can install ripgrep from the official ports:

$ doas pkg_add ripgrep

If you're a NetBSD user, then you can install ripgrep from pkgsrc:

# pkgin install ripgrep

If you're a Rust programmer, ripgrep can be installed with cargo.

  • Note that the minimum supported version of Rust for ripgrep is 1.28.0, although ripgrep may work with older versions.
  • Note that the binary may be bigger than expected because it contains debug symbols. This is intentional. To remove debug symbols and therefore reduce the file size, run strip on the binary.
$ cargo install ripgrep

When compiling with Rust 1.27 or newer, this will automatically enable SIMD optimizations for search.

ripgrep isn't currently in any other package repositories. I'd like to change that.

Building

ripgrep is written in Rust, so you'll need to grab a Rust installation in order to compile it. ripgrep compiles with Rust 1.28.0 (stable) or newer. In general, ripgrep tracks the latest stable release of the Rust compiler.

To build ripgrep:

$ git clone https://github.com/BurntSushi/ripgrep
$ cd ripgrep
$ cargo build --release
$ ./target/release/rg --version
0.1.3

If you have a Rust nightly compiler and a recent Intel CPU, then you can enable additional optional SIMD acceleration like so:

RUSTFLAGS="-C target-cpu=native" cargo build --release --features 'simd-accel avx-accel'

If your machine doesn't support AVX instructions, then simply remove avx-accel from the features list. Similarly for SIMD (which corresponds roughly to SSE instructions).

The simd-accel and avx-accel features enable SIMD support in certain ripgrep dependencies (responsible for counting lines and transcoding). They are not necessary to get SIMD optimizations for search; those are enabled automatically. Hopefully, some day, the simd-accel and avx-accel features will similarly become unnecessary.

Finally, optional PCRE2 support can be built with ripgrep by enabling the pcre2 feature:

$ cargo build --release --features 'pcre2'

(Tip: use --features 'pcre2 simd-accel avx-accel' to also include compile time SIMD optimizations, which will only work with a nightly compiler.)

Enabling the PCRE2 feature works with a stable Rust compiler and will attempt to automatically find and link with your system's PCRE2 library via pkg-config. If one doesn't exist, then ripgrep will build PCRE2 from source using your system's C compiler and then statically link it into the final executable. Static linking can be forced even when there is an available PCRE2 system library by either building ripgrep with the MUSL target or by setting PCRE2_SYS_STATIC=1.

ripgrep can be built with the MUSL target on Linux by first installing the MUSL library on your system (consult your friendly neighborhood package manager). Then you just need to add MUSL support to your Rust toolchain and rebuild ripgrep, which yields a fully static executable:

$ rustup target add x86_64-unknown-linux-musl
$ cargo build --release --target x86_64-unknown-linux-musl

Applying the --features flag from above works as expected.

Running tests

ripgrep is relatively well-tested, including both unit tests and integration tests. To run the full test suite, use:

$ cargo test --all

from the repository root.