1
0
mirror of https://github.com/BurntSushi/ripgrep.git synced 2024-12-12 19:18:24 +02:00
Commit Graph

134 Commits

Author SHA1 Message Date
Ralf Jung
d352b79294 Add new -M/--max-columns option.
This permits setting the maximum line width with respect to the number
of bytes in a line. Omitted lines (whether part of a match, replacement
or context) are replaced with a message stating that the line was
elided.

Fixes #129
2017-03-12 21:21:28 -04:00
Andrew Gallant
4ef4818130 No line numbers when searching only stdin.
This changes the default behavior of ripgrep to *not* show line numbers
when it is printing to a tty and is only searching stdin.

Fixes #380

[breaking-change]
2017-03-12 20:21:40 -04:00
Andrew Gallant
8bbe58d623 Add support for additional text encodings.
This includes, but is not limited to, UTF-16, latin-1, GBK, EUC-JP and
Shift_JIS. (Courtesy of the `encoding_rs` crate.)

Specifically, this feature enables ripgrep to search files that are
encoded in an encoding other than UTF-8. The list of available encodings
is tied directly to what the `encoding_rs` crate supports, which is in
turn tied to the Encoding Standard. The full list of available encodings
can be found here: https://encoding.spec.whatwg.org/#concept-encoding-get

This pull request also introduces the notion that text encodings can be
automatically detected on a best effort basis. Currently, the only
support for this is checking for a UTF-16 bom. In all other cases, a
text encoding of `auto` (the default) implies a UTF-8 or ASCII
compatible source encoding. When a text encoding is otherwise specified,
it is unconditionally used for all files searched.

Since ripgrep's regex engine is fundamentally built on top of UTF-8,
this feature works by transcoding the files to be searched from their
source encoding to UTF-8. This transcoding only happens when:

1. `auto` is specified and a non-UTF-8 encoding is detected.
2. A specific encoding is given by end users (including UTF-8).

When transcoding occurs, errors are handled by automatically inserting
the Unicode replacement character. In this case, ripgrep's output is
guaranteed to be valid UTF-8 (excluding non-UTF-8 file paths, if they
are printed).

In all other cases, the source text is searched directly, which implies
an assumption that it is at least ASCII compatible, but where UTF-8 is
most useful. In this scenario, encoding errors are not detected. In this
case, ripgrep's output will match the input exactly, byte-for-byte.

This design may not be optimal in all cases, but it has some advantages:

1. In the happy path ("UTF-8 everywhere") remains happy. I have not been
   able to witness any performance regressions.
2. In the non-UTF-8 path, implementation complexity is kept relatively
   low. The cost here is transcoding itself. A potentially superior
   implementation might build decoding of any encoding into the regex
   engine itself. In particular, the fundamental problem with
   transcoding everything first is that literal optimizations are nearly
   negated.

Future work should entail improving the user experience. For example, we
might want to auto-detect more text encodings. A more elaborate UX
experience might permit end users to specify multiple text encodings,
although this seems hard to pull off in an ergonomic way.

Fixes #1
2017-03-12 19:54:48 -04:00
Marc Tiehuis
adff43fbb4 Remove clap validator + add max-filesize integration tests 2017-03-08 10:17:18 -05:00
tiehuis
714ae82241 Add --max-filesize option to cli
The --max-filesize option allows filtering files which are larger than
the specified limit. This is potentially useful if one is attempting to
search a number of large files without common file-types/suffixes.

See #369.
2017-03-08 10:17:18 -05:00
Andrew Gallant
525b278049 Don't parses regexes with --files.
When the --files flag is given, ripgrep would still try to parse some of
the positional arguments as regexes. Don't do that.

Fixes #326
2017-02-18 15:34:54 -05:00
Peter Williams
22cb644eb6 termcolor: add support for output to standard error
This is essentially a rename of the existing `Stdout` type to `StandardStream`
and a change of its constructor from a single `new()` function to have two
`stdout()` and `stderr()` functions.

Under the hood, we add add internal IoStandardStream{,Lock} enums that allow
us to abstract between Stdout and Stderr conveniently. The rest of the needed
changes then fall out fairly naturally.

Fixes #324.

[breaking-change]
2017-02-09 20:57:23 -05:00
Andrew Gallant
f5a2d022ec Replace internal atty module with atty crate.
This removes all use of explicit unsafe in ripgrep proper except for
one: accessing the contents of a memory map. (Which may never go away.)
2017-01-15 16:32:30 -05:00
Andrew Gallant
a7d0e40668 Use basic SGR sequences when possible.
In Emacs, its terminal apparently doesn't support "extended" sets of
foreground/background colors. Unless we need to set an "intense" color,
we should instead use one of the eight basic color codes.

Also, remove the "intense" setting from the default set of colors. It
doesn't do much anyway and enables the default color settings to work
in Emacs out of the box.

Fixes #182 (again)
2017-01-13 19:03:03 -05:00
Andrew Gallant
7a951f103a Make --column imply --line-number.
Closes #243
2017-01-11 18:53:35 -05:00
Andrew Gallant
8751e55706 Add --path-separator flag.
This flag permits setting the path separator used for all file paths
printed by ripgrep in normal operation.

Fixes #275
2017-01-10 18:16:15 -05:00
Andrew Gallant
461e0c4e33 Don't search stdout redirected file.
When running ripgrep like this:

    rg foo > output

we must be careful not to search `output` since ripgrep is actively writing
to it. Searching it can cause massive blowups where the file grows without
bound.

While this is conceptually easy to fix (check the inode of the redirection
and the inode of the file you're about to search), there are a few problems
with it.

First, inodes are a Unix thing, so we need a Windows specific solution to
this as well. To resolve this concern, I created a new crate, `same-file`,
which provides a cross platform abstraction.

Second, stat'ing every file is costly. This is not avoidable on Windows,
but on Unix, we can get the inode number directly from directory traversal.
However, this information wasn't exposed, but now it is (through both the
ignore and walkdir crates).

Fixes #286
2017-01-09 16:12:08 -05:00
Andrew Gallant
b65a8c353b Add --sort-files flag.
When used, parallelism is disabled but the results are sorted by file
path.

Closes #263
2017-01-06 22:43:59 -05:00
Andrew Gallant
b187c1a817 Rejigger bold and intense settings.
Previously, ripgrep would only emit the 'bold' ANSI escape sequence if
no foreground or background color was set. Instead, it would convert colors
to their "intense" versions if bold was set. The intent was to do the same
thing on Windows and Unix. However, this had a few negative side effects:

  1. Omitting the 'bold' ANSI escape when 'bold' was set is surprising.
  2. Intense colors can look quite bad and be hard to read.

To fix this, we introduce a new setting called 'intense' in the --colors
flag, and thread that down through to the public API of the `termcolor`
crate. The 'intense' setting has environment specific behavior:

  1. In ANSI mode, it will convert the selected color to its "intense"
     variant.
  2. In the Windows console, it will make the text "intense."

There is no longer any "smart" handling of the 'bold' style. The 'bold'
ANSI escape is always emitted when it is selected. In the Windows
console, the 'bold' setting now has no effect. Note that this is a
breaking change.

Fixes #266, #293
2017-01-06 20:09:51 -05:00
Andrew Gallant
163e00677a Update to regex 0.2. 2017-01-01 01:03:21 -05:00
Andrew Gallant
de5cb7d22e Remove special ^C handling.
This means that ripgrep will no longer try to reset your colors in your
terminal if you kill it while searching. This could result in messing up
the colors in your terminal, and the fix is to simply run some other
command that resets them for you. For example:

    $ echo -ne "\033[0m"

The reason why the ^C handling was removed is because it is irrevocably
broken on Windows and is impossible to do correctly and efficiently in
ANSI terminals.

Fixes #281
2016-12-24 12:53:09 -05:00
Andrew Gallant
084d3f4911 Small code cleanups. 2016-12-24 10:06:37 -05:00
Leonardo Yvens
dd5ded2f78 fix some clippy lints (#288) 2016-12-23 14:53:35 -05:00
Andrew Gallant
d06f84ced3 Get rid of special mmap decision on Windows.
I spent some quality time on my Windows 10 laptop and it appears to
suffer from a similar trade-off as on Linux: mmaps are bad for large
directory traversals but good for single large files.

Darwin continues to reject memory maps in all cases (unless explicitly
requested), but more testing should be done there.
2016-11-20 15:32:50 -05:00
Andrew Gallant
9598331fa8 Propagate no_messages option to worker.
Fixes #241
2016-11-20 15:01:37 -05:00
Andrew Gallant
e8a30cb893 Completely re-work colored output and tty handling.
This commit completely guts all of the color handling code and replaces
most of it with two new crates: wincolor and termcolor. wincolor
provides a simple API to coloring using the Windows console and
termcolor provides a platform independent coloring API tuned for
multithreaded command line programs. This required a lot more
flexibility than what the `term` crate provided, so it was dropped.
We instead switch to writing ANSI escape sequences directly and ignore
the TERMINFO database.

In addition to fixing several bugs, this commit also permits end users
to customize colors to a certain extent. For example, this command will
set the match color to magenta and the line number background to yellow:

    rg --colors 'match:fg:magenta' --colors 'line:bg:yellow' foo

For tty handling, we've adopted a hack from `git` to do tty detection in
MSYS/mintty terminals. As a result, ripgrep should get both color
detection and piping correct on Windows regardless of which terminal you
use.

Finally, switch to line buffering. Performance doesn't seem to be
impacted and it's an otherwise more user friendly option.

Fixes #37, Fixes #51, Fixes #94, Fixes #117, Fixes #182, Fixes #231
2016-11-20 11:14:52 -05:00
Andrew Gallant
03f7605322 Rename --files-without-matches to --files-without-match.
This is to be consistent with grep.
2016-11-19 20:15:41 -05:00
Daniel Luz
bd3e7eedb1 Add --files-without-matches flag.
Performs the opposite of --files-with-matches: only shows paths of
files that contain zero matches.

Closes #138
2016-11-19 21:48:59 -02:00
Andrew Gallant
0302d58eb8 Fix stdin bug with --file.
When `rg -f-` is used, the default search path should be `./` and not
`-`.
2016-11-17 20:48:11 -05:00
Andrew Gallant
92dc402f7f Switch from Docopt to Clap.
There were two important reasons for the switch:

1. Performance. Docopt does poorly when the argv becomes large, which is
   a reasonable common use case for search tools. (e.g., use with xargs)
2. Better failure modes. Clap knows a lot more about how a particular
   argv might be invalid, and can therefore provide much clearer error
   messages.

While both were important, (1) made it urgent.

Note that since Clap requires at least Rust 1.11, this will in turn
increase the minimum Rust version supported by ripgrep from Rust 1.9 to
Rust 1.11. It is therefore a breaking change, so the soonest release of
ripgrep with Clap will have to be 0.3.

There is also at least one subtle breaking change in real usage.
Previous to this commit, this used to work:

    rg -e -foo

Where this would cause ripgrep to search for the string `-foo`. Clap
currently has problems supporting this use case
(see: https://github.com/kbknapp/clap-rs/issues/742),
but it can be worked around by using this instead:

    rg -e [-]foo

or even

    rg [-]foo

and this still works:

    rg -- -foo

This commit also adds Bash, Fish and PowerShell completion files to the
release, fixes a bug that prevented ripgrep from working on file
paths containing invalid UTF-8 and shows short descriptions in the
output of `-h` but longer descriptions in the output of `--help`.

Fixes #136, Fixes #189, Fixes #210, Fixes #230
2016-11-17 19:53:41 -05:00
Eric Kidd
e9cd0a1cc3 Allow specifying patterns with -f FILE and -f-
This is a somewhat basic implementation of `-f-` (#7), with unit tests.
Changes include:

1. The internals of the `pattern` function have been refactored to avoid
   code duplication, but there's a lot more we could do.  Right now we
   read the entire pattern list into a `Vec`.
2. There's now a `WorkDir::pipe` command that allows sending standard
   input to `rg` when testing.

Not implemented: aho-corasick.
2016-11-15 13:00:16 -05:00
Andrew Gallant
5b73dcc8ab Rework parallelism in directory iterator.
Previously, ignore::WalkParallel would invoke the callback for all
*explicitly* given file paths in a single thread, which effectively
meant that `rg pattern foo bar baz ...` didn't actually search foo, bar
and baz in parallel.

The code was structured that way to avoid spinning up workers if no
directory paths were given. The original intention was probably to have
a separate pool of threads responsible for searching, but ripgrep ended
up just reusing the ignore::WalkParallel workers themselves for searching,
and thereby subjected to its sub-par performance in this case.

The code has been restructured so that file paths are sent to the workers,
which brings back parallelism.

Fixes #226
2016-11-09 17:19:40 -05:00
Andrew Gallant
77ad7588ae Add --no-messages flag.
This flag is similar to what's found in grep: it will suppress all error
messages, such as those shown when a particular file couldn't be read.

Closes #149
2016-11-06 14:36:08 -05:00
Andrew Gallant
58aca2efb2 Add -m/--max-count flag.
This flag limits the number of matches printed *per file*.

Closes #159
2016-11-06 13:09:53 -05:00
Andrew Gallant
277dda544c Include the name "ripgrep" in more places.
Fixes #203
2016-11-06 12:21:36 -05:00
Andrew Gallant
598b162fea Note -e/--regexp's additional usefulness.
Specifically, it can be used when searching for patterns that start
with a dash.

Fixes #215
2016-11-06 12:10:27 -05:00
Andrew Gallant
b272be25fa Add parallel recursive directory iterator.
This adds a new walk type in the `ignore` crate, `WalkParallel`, which
provides a way for recursively iterating over a set of paths in parallel
while respecting various ignore rules.

The API is a bit strange, as a closure producing a closure isn't
something one often sees, but it does seem to work well.

This also allowed us to simplify much of the worker logic in ripgrep
proper, where MultiWorker is now gone.
2016-11-05 21:45:55 -04:00
Andrew Gallant
d79add341b Move all gitignore matching to separate crate.
This PR introduces a new sub-crate, `ignore`, which primarily provides a
fast recursive directory iterator that respects ignore files like
gitignore and other configurable filtering rules based on globs or even
file types.

This results in a substantial source of complexity moved out of ripgrep's
core and into a reusable component that others can now (hopefully)
benefit from.

While much of the ignore code carried over from ripgrep's core, a
substantial portion of it was rewritten with the following goals in
mind:

1. Reuse matchers built from gitignore files across directory iteration.
2. Design the matcher data structure to be amenable for parallelizing
   directory iteration. (Indeed, writing the parallel iterator is the
   next step.)

Fixes #9, #44, #45
2016-10-29 20:48:59 -04:00
Alex Burka
4c3025ab1c clarify docs for --threads 2016-10-11 17:32:51 -04:00
Andrew Gallant
a3537aa32a Update darwin cfg attributes. 2016-10-10 21:48:47 -04:00
Andrew Gallant
60c016c243 Fix docopt usage string. Gah. 2016-10-10 20:49:39 -04:00
Andrew Gallant
4665128f25 Clarify documentation for --replace.
Also add a minor clarification for --type-add.

Fixes #147
2016-10-10 20:19:45 -04:00
Andrew Gallant
e8645dc8ae style nits 2016-10-10 19:27:12 -04:00
Ian Kerins
1c964372ad Always follow symlinks on explicit file arguments. 2016-10-08 22:40:03 -04:00
Andrew Gallant
cee2f09a6d Merge pull request #121 from lilydjwg/master
if --color always, always print with color, even when --vimgrep is given
2016-09-29 16:49:44 -04:00
Andrew Gallant
46dff8f4be Be better with short circuiting with --quiet.
It didn't make sense for --quiet to be part of the printer, because --quiet
doesn't just mean "don't print," it also means, "stop after the first
match is found." This needs to be wired all the way up through directory
traversal, and it also needs to cause all of the search workers to quit
as well. We do it with an atomic that is only checked with --quiet is
given.

Fixes #116.
2016-09-28 20:50:50 -04:00
Andrew Gallant
925d0db9f0 Add -s/--case-sensitive flag.
This flag overrides both --smart-case and --ignore-case.

Closes #124.
2016-09-28 16:32:29 -04:00
依云
5943b1effe if --color always, always print with color, even when --vimgrep is given 2016-09-28 20:10:07 +08:00
Garrett Squire
babe80d498 add a max-depth option for directory traversal
CR and add integration test
2016-09-27 16:14:53 -07:00
Andrew Gallant
7a3fd1f23f Add a --null flag.
This flag causes a NUL byte to follow any file path in ripgrep's output.

Closes #89.
2016-09-26 19:21:17 -04:00
Andrew Gallant
2da0eab2b8 Don't initialize ignores for file arguments.
We'll never use them, so it's wasted effort.
2016-09-26 18:44:19 -04:00
Andrew Gallant
f85822266f Don't use an intermediate buffer when --threads=1.
Fixes #8
2016-09-25 21:27:17 -04:00
Andrew Gallant
982265af70 Move --files-with-matches to less common options. 2016-09-25 18:32:41 -04:00
Andrew Gallant
9dc5464c84 Stop after first match is found with --quiet.
Fixes #77.
2016-09-25 15:01:29 -04:00
Andrew Gallant
95edcd4d3a Merge pull request #42 from andschwa/files-with-matches
Files with matches
2016-09-25 14:53:31 -04:00
Andrew Gallant
d97f404970 Stupid docopt.
It thinks `--type-clear is` is a flag spec.
2016-09-25 14:47:35 -04:00
Andrew Gallant
b2bbd46178 Clarify documentation of --type-add.
This explains it a bit more based on end user feedback. We also fix
the example, which was wrong.

Fixes #82.
2016-09-25 14:37:01 -04:00
Andrew Gallant
ab0d1c1c79 Be more conservative with stdin.
If no paths are given to ripgrep, only read from stdin if it's a file or
a FIFO. In particular, if something like `rg foo < /dev/null` is used,
then don't try to read from stdin.

Fixes #35, #81
2016-09-25 11:14:54 -04:00
Andrew Schwartzmeyer
a8f3d9e87e Add --files-with-matches flag.
Closes #26.

Acts like --count but emits only the paths of files with matches,
suitable for piping to xargs. Both mmap and no-mmap searches terminate
after the first match is found. Documentation updated and tests added.
2016-09-24 21:40:17 -07:00
Andrew Gallant
1595f0faf5 Add --smart-case.
It does what it says on the tin.

Closes #70.
2016-09-24 21:51:04 -04:00
Andrew Gallant
8eeb0c0b60 Add --no-ignore-vcs flag.
This flag will respect .ignore but not .gitignore.

Closes #68.
2016-09-24 21:31:24 -04:00
Andrew Gallant
423f2a1927 Permit options with --help/--version.
Fixes #47.
2016-09-24 21:13:24 -04:00
Andrew Gallant
a44735aa87 Tweak memory maps on darwin.
Namely, don't automatically pick memory maps on darwin, ever. They
appear slower than standard read calls.

Closes #36.
2016-09-24 20:48:05 -04:00
Andrew Gallant
c8227e0cf3 Don't ignore first path when using --files.
This is a docopt oddity, but probably not a bug. If --files is given,
then just interpret the pattern (if not empty) as the first file path.

Fixes #64.
2016-09-24 20:22:02 -04:00
Andrew Gallant
ce85df1d2e Clarify what rg does in --help.
Fixes #24.
2016-09-24 19:26:28 -04:00
Andrew Gallant
a6e3cab65a Add --no-filename flag.
When this flag is set, a filename is never shown for a match.

Closes #20
2016-09-24 19:24:24 -04:00
Andrew Gallant
9ce0484670 Clarify the documentation of the --type-* flags.
Fixes #15
2016-09-24 18:55:48 -04:00
Andrew Gallant
cc90511ab2 Switch from .rgignore to .ignore.
But don't actually remove support for .rgignore until the next semver
bump.

Note that this puts us in line with the silver searcher:
https://github.com/ggreer/the_silver_searcher/pull/974

Fixes #40
2016-09-23 22:44:33 -04:00
Andrew Gallant
dfebed6cbe Add --vimgrep flag.
The --vimgrep flag forces a line to be printed for every match, with
line and column numbers.
2016-09-22 21:32:38 -04:00
Andrew Gallant
5a9883d27c Try to use memory maps more aggressively on Windows.
Some brief playing around suggests that it is faster.

However, it's probably slower in a VM. Let's prioritize native users.
2016-09-21 20:47:40 -04:00
Andrew Gallant
7402db7b43 Add "unrestricted" flag.
I don't like having multiple flags do the same thing, but -u, -uu and -uuu
are much easier to remember, particularly with -uuu meaning "search
everything."
2016-09-20 20:24:03 -04:00
Andrew Gallant
f7ee914dd3 Add support for searching multiple patterns with -e.
Also, change -Q/--literal to -F/--fixed-strings because compatibility
with grep is probably better.
2016-09-17 16:55:58 -04:00
Andrew Gallant
bfbbfbf979 fix windows build
Why isn't CI running on each push? It seems to only be running on tagged
commits.
2016-09-17 12:54:46 -04:00
Andrew Gallant
92c918ebd9 --no-ignore implies --no-ignore-parent 2016-09-14 14:33:37 -04:00
Andrew Gallant
fdca74148d Stream results when feasible.
For example, when only a single file (or stdin) is being searched, then we
should be able to print directly to the terminal instead of intermediate
buffers. (The buffers are only necessary for parallelism.)

Closes #4.
2016-09-13 21:11:46 -04:00
Andrew Gallant
68fa50709e Don't skip the first arg.
Docopt will do it for us.
2016-09-11 13:33:25 -04:00
Andrew Gallant
ab91e4b874 Don't panic when an argument is invalid UTF-8.
Suggest a workaround.
2016-09-11 13:27:08 -04:00
Andrew Gallant
2b943eda47 Make file type filtering a lot faster.
We do this by avoiding using a RegexSet (*sigh*). In particular, file
type matching has much simpler semantics than gitignore files, so we don't
actually need to care which file type matched. Therefore, we can get away
with a single regex with a giant alternation.
2016-09-11 13:26:53 -04:00
Andrew Gallant
cb0f8fd2fa Bump default thread count to 8. 2016-09-11 00:42:39 -04:00
Andrew Gallant
e3da726836 Rename search module to search_stream.
The name better reflects the difference between it and the search_buffer
module.
2016-09-10 00:08:42 -04:00
Andrew Gallant
5b36c86c15 Rejigger the atty detection stuff. 2016-09-10 00:05:20 -04:00
Andrew Gallant
f83cd63b11 Add integration tests. 2016-09-09 22:58:30 -04:00
Andrew Gallant
0766617e07 Refactor how coloring is done.
All in the name of appeasing Windows.
2016-09-08 21:46:14 -04:00
Andrew Gallant
a744ec133d Rename xrep to ripgrep. 2016-09-08 16:15:44 -04:00
Andrew Gallant
ca058d7584 Add support for memory maps.
I though plain `read` had usurped them, but when searching a very small
number of files, mmaps can be around 20% faster on Linux. It'd be really
unfortunate to leave that on the table.

Mmap searching doesn't support contexts yet, but we probably don't really
care. And duplicating that logic doesn't sound fun. Without contexts, mmap
searching is delightfully simple.
2016-09-06 21:47:33 -04:00
Andrew Gallant
5938bed339 Add support for printing column numbers. 2016-09-06 19:50:27 -04:00
Andrew Gallant
9948e0ca07 Only create the Grep searcher once. 2016-09-06 19:33:19 -04:00
Andrew Gallant
7a149c20fe More progress. With coloring! 2016-09-05 17:36:41 -04:00
Andrew Gallant
812cdb13c6 Lots of progress:
- Refactored interaction between CLI args and rest of xrep.
  - Filling in a lot more options, including file type filtering.
  - Fixing some bugs in globbing/ignoring.
  - More documentation.
2016-09-05 00:52:23 -04:00