1
0
mirror of https://github.com/BurntSushi/ripgrep.git synced 2024-12-12 19:18:24 +02:00
Commit Graph

139 Commits

Author SHA1 Message Date
Andrew Gallant
229d1a8d41
cli: fix arbitrary execution of program bug
This fixes a bug only present on Windows that would permit someone to
execute an arbitrary program if they crafted an appropriate directory
tree. Namely, if someone put an executable named 'xz.exe' in the root of
a directory tree and one ran 'rg -z foo' from the root of that tree,
then the 'xz.exe' executable in that tree would execute if there are any
'xz' files anywhere in the tree.

The root cause of this problem is that 'CreateProcess' on Windows will
implicitly look in the current working directory for an executable when
it is given a relative path to a program. Rust's standard library allows
this behavior to occur, so we work around it here. We work around it by
explicitly resolving programs like 'xz' via 'PATH'. That way, we only
ever pass an absolute path to 'CreateProcess', which avoids the implicit
behavior of checking the current working directory.

This fix doesn't apply to non-Windows systems as it is believed to only
impact Windows. In theory, the bug could apply on Unix if '.' is in
one's PATH, but at that point, you reap what you sow.

While the extent to which this is a security problem isn't clear, I
think users generally expect to be able to download or clone
repositories from the Internet and run ripgrep on them without fear of
anything too awful happening. Being able to execute an arbitrary program
probably violates that expectation. Therefore, CVE-2021-3013[1] was
created for this issue.

We apply the same logic to the --pre command, since the --pre command is
likely in a user's config file and it would be surprising for something
that the user is searching to modify which preprocessor command is used.

The --pre and -z/--search-zip flags are the only two ways that ripgrep
will invoke external programs, so this should cover any possible
exploitable cases of this bug.

[1] - https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2021-3013
2021-05-29 09:36:48 -04:00
Andrew Gallant
581a35e568
impl: fix --multiline anchored match bug
This fixes a bug where using \A or (?-m)^ in combination with
-U/--multiline would permit matches that aren't anchored to the
beginning of the file. The underlying cause was an optimization that
occurred when mmaps couldn't be used. Namely, ripgrep tries to still
read the input incrementally if it knows the pattern can't match through
a new line. But the detection logic was flawed, since it didn't account
for line anchors. This commit fixes that.

Fixes #1878, Fixes #1879
2021-05-29 07:37:28 -04:00
jack1142
ba965962fe
ignore/types: add po files to supported types
See: https://www.gnu.org/software/gettext/manual/html_node/PO-Files.html

Closes #1875
2021-05-28 12:06:10 -04:00
Andrew Gallant
94e4b8e301
printer: fix --vimgrep for multi-line mode
It turned out that --vimgrep wasn't quite getting the column of each
match correctly. Instead of printing column numbers relative to the
current line, it was printing column numbers as byte offsets relative to
where the match began. To fix this, we simply subtract the offset of the
line number from the beginning of the match. If the beginning of the
match came before the start of the current line, then there's really
nothing sensible we can do other than to use a column number of 1, which
we now document.

Interestingly, existing tests were checking that the previous behavior
was intended. My only defense is that I somehow tricked myself into
thinking it was a byte offset instead of a column number.

Kudos to @bfrg for calling this out in #1866:
https://github.com/BurntSushi/ripgrep/issues/1866#issuecomment-841635553
2021-05-15 08:27:59 -04:00
Alessandro Caputo
2af77242c5
doc: fix typo in --engine flag docs
Fixes #1862
2021-05-08 15:35:44 -04:00
Dan Bjorge
92286ad4d2
doc: clarify --hidden definition
On Windows, we didn't previously document that ripgrep
respected both the prefix-dot convention _and_ the "hidden"
attribute on files.

Fixes #1847
2021-04-15 19:21:26 -04:00
jgart
4ebe8375ec
ignore/types: add mint
PR #1844
2021-04-04 08:00:12 -04:00
Andrew Gallant
7923d25228
core: add a 'trace' message
This message will emit the binary detection mechanism being used for
each file.

This does not noticeably increases the number of log messages, as the
'trace' level is already used for emitting messages for every file
searched.

This trace message was added in the course of investigating #1838.
2021-03-31 13:54:00 -04:00
aricha1940
1c3eebefec
searcher: update outdated comment for buffer size
Looks like this was accidentally left set to 8 in commit 46fb77c.

PR #1839
2021-03-31 08:18:38 -04:00
Andrew Gallant
64ac2ebe0f
tests: fix tests for buffer size change
Sadly, there were several tests that are coupled to the size of the
buffer used by ripgrep. Making the tests agnostic to the size is
difficult. And it's annoying to fix the tests. But we rarely change the
buffer size, so ¯\_(ツ)_/¯.
2021-03-23 18:14:18 -04:00
Andrew Gallant
46fb77c20c
searcher: bump buffer size
This increases the initial buffer size from 8KB to 64KB. This actually
leads to a reasonably noticeable improvement in at least one work-load,
and is unlikely to regress in any other case. Also, since Rust programs
(at least on Linux) seem to always use a minimum of 6-8MB of memory,
adding an extra 56KB is negligible.

Before:

    $ hyperfine -i "rg 'zqzqzqzq' OpenSubtitles2018.raw.en --no-mmap"
    Benchmark #1: rg 'zqzqzqzq' OpenSubtitles2018.raw.en --no-mmap
      Time (mean ± σ):      2.109 s ±  0.012 s    [User: 565.5 ms, System: 1541.6 ms]
      Range (min … max):    2.094 s …  2.128 s    10 runs

After:

    $ hyperfine -i "rg 'zqzqzqzq' OpenSubtitles2018.raw.en --no-mmap"
    Benchmark #1: rg 'zqzqzqzq' OpenSubtitles2018.raw.en --no-mmap
      Time (mean ± σ):      1.802 s ±  0.006 s    [User: 462.3 ms, System: 1337.9 ms]
      Range (min … max):    1.795 s …  1.814 s    10 runs
2021-03-23 17:45:02 -04:00
Sergei Vorobev
9c8d873a75
ignore/types: improve bazel globs
Adds *.BUILD and *.bazelrc.

PR #1789
2021-01-30 18:22:48 -05:00
Andrew Gallant
7899a4b931
regex: s/CachedThreadLocal/ThreadLocal
CachedThreadLocal has been deprecated. We bump thread_local's minimal
version corresponding to that deprecation as well.
2021-01-25 10:38:05 -05:00
Andrew Gallant
3a1780d841
deps: replace memmap with memmap2
memmap is unmaintained at this point and it is being flagged as a
RUSTSEC advisory in ripgrep. This doesn't seem like that big of a deal
to me honestly, but memmap2 looks like a fine choice at this point.

Fixes #1785, Closes #1786
2021-01-17 18:49:51 -05:00
Andrew Gallant
a6d05475fb
ignore-0.4.17 2020-11-23 10:25:33 -05:00
Roey Darwish Dror
020c5453a5
cli: fix stdin detection for Powershell on Unix
It seems that PowerShell uses sockets instead of FIFOs to redirect the
output between commands. So add `is_socket` to our `is_readable_stdin`
check.

This seems unlikely to cause problems and it probably more generally
correct than what we had before. In theory, it could cause problems if
it produces false positives, in which case, ripgrep will try to read
stdin when it should search the current working directory. (And this
usually winds up manifesting as ripgrep blocking forever.) But, if the
stdin handle reports itself as a socket, then it seems like we should
read it.

Fixes #1741, Closes #1742
2020-11-23 10:23:34 -05:00
Ed Page
873abecbf1
ignore: provide underlying IO Error
`ignore::Error` wraps `std::io::Error` with additional information
(as well as expose non-IO errors). For people wanting to inspect what
the error is, they have to recursively match the Enum. This provides
`io_error` and `into_io_error` helpers to do this for the user.

PR #1740
2020-11-23 10:19:31 -05:00
James Harr
44e69ba627
ignore/types: add yang file type
YANG is described in RFC 6020
https://tools.ietf.org/html/rfc6020

PR #1736
2020-11-20 09:41:29 -05:00
Andrew Gallant
d97fb72d84
doc: update CI links in crate READMEs
I switched to GitHub Actions long ago, which replaces both Travis and
AppVeyor.

Fixes #1732
2020-11-16 19:07:16 -05:00
Andrew Gallant
d6365117e2
doc: sync --help output with man page
The man page had the correct usage hints, but the -h/--help output was
using an older more incorrect version of the hints.

Closes #1730 (again)
2020-11-15 15:27:23 -05:00
Alex Touchet
3ca324fda7
doc: update several links to use https
PR #1724
2020-11-03 10:33:36 -05:00
Andrew Gallant
2819212f89 printer: tweak binary detection message format
This roughly matches similar changes made in GNU grep recently.
2020-11-02 10:52:51 -05:00
Andrew Gallant
810be0b348 deps: update base64 to 0.13.0 2020-11-02 10:52:51 -05:00
Andrew Gallant
3ef63dacbe deps: targeted update of some dependencies
This updates encoding_rs, crossbeam-utils and crossbeam-channel. This
serves two purposes. The encoding_rs update fixes a compilation failure
on the latest nightly. The crossbeam updates are good sense and to
reduce duplicate dependencies such as cfg-if. (Although, we note that
the log crate still pulls in cfg-if 0.1, so ripgrep has a duplicate
dependency there for now. But it's very small.)

Fixes #1721, Closes #1705
2020-11-02 10:52:51 -05:00
Vanessa McHale
e1ac18ef06
ignore/types: add Futhark
See: https://futhark-lang.org/

PR #1720
2020-10-31 12:10:15 -04:00
Brandon Adams
ba3f9673ad
ignore/types: generalize bazel type a bit
Bazel supports `BUILD.bazel` as well as `WORKSPACE.bazel`. In
addition, it is common to ship BUILD/WORKSPACE templates for
external repositories suffixed with .bazel for easier tool
recognition.

Co-authored-by: Brandon Adams <brandon.adams@imc.com>

PR #1716
2020-10-23 12:24:30 -04:00
Andrew Gallant
c777e2cd57
globset-0.4.6 2020-10-21 21:10:43 -04:00
Ajeet D'Souza
e5639cf22d
globset: remove regex unicode dependency
Since the translation from a glob to a regex always
disables Unicode in the regex, it follows that we shouldn't
need regex's Unicode features enabled.

Now, ripgrep enables Unicode features in its regex
dependency and of course uses them, which will cause
globset to have it enabled in the ripgrep build as well. So
this doesn't actually change anything for ripgrep. But this
does slim thing downs for folks using globset independently
of ripgrep.

PR #1712
2020-10-19 14:29:05 -04:00
Dương Đỗ Minh Châu
86c843a44b
ignore/types: add a type for minified files
Fixes #1710, PR #1711
2020-10-19 09:10:54 -04:00
Andrew Gallant
2b1637d1db
doc: clarify how -S/--smart-case works
Whether or not smart case kicks in can be a little subtle in some cases.
So we document the specific conditions in which it applies. These
conditions were taken directly from the public API docs of the
`grep-regex` crate:
https://docs.rs/grep-regex/0.1.8/grep_regex/struct.RegexMatcherBuilder.html#method.case_smart

Fixes #1708
2020-10-17 18:55:44 -04:00
Andrew Pyatkov
6301e20ee4
ignore/types: add flatbuffers type
See: https://google.github.io/flatbuffers/

PR #1707
2020-10-16 20:19:16 -04:00
dana
145cef2eff
doc: elaborate on the function of -u/--unrestricted
Fixes #1703
2020-10-16 09:52:42 -04:00
Andy Freeland
fc2a99bb1f
ignore/types: add vcl (#1659)
VCL is the Varnish Configuration Language used by Varnish and Fastly.

https://varnish-cache.org/docs/trunk/users-guide/vcl.html

PR #1659
2020-08-19 16:28:14 -04:00
Raimon Grau (rgrau)
ffd4c9ccba
ignore/types: add racket
PR #1628
2020-06-25 08:51:32 -04:00
jtrakk
a16bfcb3d6
ignore/types: add dvc
This provides support for DVC files (https://dvc.org/).

PR #1608
2020-06-09 07:44:09 -04:00
Martin Michlmayr
1b2c1dc675
doc: fix typos
PR #1605
2020-06-04 09:06:09 -04:00
Andrew Gallant
f97cc623f7
grep-0.2.7 2020-05-29 09:17:24 -04:00
Andrew Gallant
f35de5c523
grep: update minimal dependency versions 2020-05-29 09:17:08 -04:00
Andrew Gallant
c9bb78ceba
grep-cli-0.1.5 2020-05-29 09:14:18 -04:00
Andrew Gallant
72bdde6771
ignore-0.4.16 2020-05-29 09:13:02 -04:00
Andy Salerno
e8822ce97a
ignore/doc: update misleading documentation
This likely originated from a bad copy/paste.

PR #1596
2020-05-24 23:12:53 -04:00
Andrew Gallant
a700b75843
doc: clarify capture group indices
And in particular, note the special $0 index, which corresponds to the
entire match.

Fixes #1591
2020-05-21 22:22:51 -04:00
Gerion Entrup
b72ad8f8aa
ignore/types: add meson filetype
Closes #1586, PR #1587
2020-05-18 14:01:35 -04:00
Andrew Gallant
1980630f17
doc: fix egregious markup output
We use '+++' syntax to output a literal '**' for a '--glob' example.
This '+++' syntax is pretty ugly when rendered literally via --help. We
fix this by hackily inserting the '+++' syntax for its one specific case
that we need it during man page generation.

Not ideal but it works. And --help still has some '*foo*' markup, but we
live with that for now.

Fixes #1581
2020-05-13 08:13:05 -04:00
Andrew Gallant
72807462e8
deps: update minimal versions for dependencies 2020-05-09 10:39:43 -04:00
Andrew Gallant
08dee094dd
grep-0.2.6 2020-05-09 10:37:29 -04:00
Andrew Gallant
caa53b7b09
grep: update minimal dependency versions 2020-05-09 10:37:08 -04:00
Andrew Gallant
c5d6141562
grep-printer-0.1.5 2020-05-09 10:33:02 -04:00
Andrew Gallant
c0f0492b98
grep-regex-0.1.8 2020-05-09 10:31:29 -04:00
Andrew Gallant
568018386b
ignore-0.4.15 2020-05-09 10:27:19 -04:00
Andrew Gallant
b458cf39f2
deps: update to base64 0.12
No code changes were necessary.
2020-05-09 10:25:37 -04:00
Casey Rodarmor
793c1179cc ignore: allow filtering with predicate
Adds `WalkBuilder::filter_entry` that takes a predicate to be applied to
all entries. If the predicate returns `false` on a given entry, that
entry and all children will be skipped.

Fixes #1555, Closes #1557
2020-05-08 23:24:40 -04:00
Wieland Hoffmann
df7a3bfc7f grep-cli: support files compressed by compress(1)
While Linux distributions (at least Arch Linux, RHEL, Debian) do not support
compressing files with compress(1), macOS & AIX do (the utility is part of
POSIX). Additionally, gzip is able to uncompress such compressed files and
provides an `uncompress` binary.

Closes #1547
2020-05-08 23:24:40 -04:00
Andrew Gallant
28f2a93cae doc: shorten -h/--help prelude
It has grown quite long. It would be nice if we could shorten this only
when -h is used and keep it long for --help, but it seems clap doesn't
let this happen. (It does have `about` and `long_about` options, but
they don't work, even when I disable the use of the template.)

The longer prelude is now only available in the man page.

This addresses #189.
2020-05-08 23:24:40 -04:00
Andrew Gallant
64a4dee495 cli: improve invalid UTF-8 pattern error message
When a pattern with invalid UTF-8 is given, the error message suggests
unqualified use of hex escape sequences to match arbitrary bytes. But
you *also* need to disable Unicode mode. So include that in the error
message.

Fixes #1339
2020-05-08 23:24:40 -04:00
Andrew Gallant
50840ea43b doc: note how to escape a '$' in --replace
Fixes #1524
2020-05-08 23:24:40 -04:00
Andrew Gallant
17dcc2bf51 doc: clarify that *files* override gitignores
This attempts to fix some mild confusion that came up as part of #1574.
Specifically:
https://github.com/BurntSushi/ripgrep/issues/1574#issuecomment-625780436
2020-05-08 23:24:40 -04:00
Andrew Gallant
9a858e4909 doc: add config file note for --type-{add,clear}
This clarifies that persistence is possible via a configuration file.

Fixes #1571
2020-05-08 23:24:40 -04:00
Andrew Gallant
7ed9a31819 printer: fix --count-matches output
In order to implement --count-matches, we simply re-execute the regex on
the spans reported by the searcher. The spans always correspond to the
lines that participated in the match. This is the correct thing to do,
except when the regex contains look-ahead (or look-behind).

In particular, the look-around permits the regex's match success to
depends on an arbitrary point before or after the lines actually
reported as participating in the match. Since only the matched lines are
reported to the printer, it is possible for subsequent searching on
those lines to fail.

A true fix for this would somehow make the total span available to the
printer. But that seems tricky since it isn't always available. For
PCRE2's case in multiline mode, it is available because we force it to
be so for correctness.

For now, we simply detect this corner case heuristically. If the match
count is zero, then it necessarily means there is some kind of
look-around that isn't matching. So we set the match count to 1. This is
probably incorrect in some cases, although my brain can't quite come up
with a concrete example. Nevertheless, this is strictly better than the
status quo.

Fixes #1573
2020-05-08 23:24:40 -04:00
Andrew Gallant
139f186e57 crates/ignore: switch to depth first traversal
This replaces the use of channels in the parallel directory traversal
with a simple stack. The primary motivation for this change is to reduce
peak memory usage. In particular, when using a channel (which is a
queue), we wind up visiting files in a breadth first fashion. Using a
stack switches us to a depth first traversal. While there are no real
intrinsic differences, depth first traversal generally tends to use less
memory because directory trees are more commonly wide than they are
deep.

In particular, the queue/stack size itself is not the only concern. In
one recent case documented in #1550, a user wanted to search all Rust
crates. The directory structure was shallow but extremely wide, with a
single directory containing all crates. This in turn results is in
descending into each of those directories and building a gitignore
matcher for each (since most crates have `.gitignore` files) before ever
searching a single file. This means that ripgrep has all such matchers
in memory simultaneously, which winds up using quite a bit of memory.

In a depth first traversal, peak memory usage is much lower because
gitignore matches are built and discarded more quickly. In the case of
searching all crates, the peak memory usage decrease is dramatic. On my
system, it shrinks by an order magnitude, from almost 1GB to 50MB. The
decline in peak memory usage is consistent across other use cases as
well, but is typically more modest. For example, searching the Linux
repo has a 50% decrease in peak memory usage and searching the Chromium
repo has a 25% decrease in peak memory usage.

Search times generally remain unchanged, although some ad hoc benchmarks
that I typically run have gotten a bit slower. As far as I can tell,
this appears to be result of scheduling changes. Namely, the depth first
traversal seems to result in searching some very large files towards the
end of the search, which reduces the effectiveness of parallelism and
makes the overall search take longer. This seems to suggest that a stack
isn't optimal. It would instead perhaps be better to prioritize
searching larger files first, but it's not quite clear how to do this
without introducing more overhead (getting the file size for each file
requires a stat call).

Fixes #1550
2020-04-18 11:33:03 -04:00
Andrew Gallant
a75b4d122a
doc: fix newline escape
Fixes #1551
2020-04-13 08:49:27 -04:00
Andrew Gallant
1c4b5adb7b
regex: fix another inner literal bug
It looks like `is_simple` wasn't quite correct.

I can't wait until this code is rewritten. It is still not quite clearly
correct to me.

Fixes #1537
2020-04-01 20:37:48 -04:00
Marius Schulz
3d6a58faff
doc: fix typo in help description
PR #1536
2020-03-30 17:31:16 -04:00
Andrew Gallant
09a4b75baf
ignore-0.4.14 2020-03-29 18:49:01 -04:00
Zoltan Puskas
4dfea016b9 ignore/types: add ebuild type
Add support for Gentoo's portage package manager spec files:
https://wiki.gentoo.org/wiki/Portage
2020-03-29 18:44:04 -04:00
Andrew Gallant
67c0f576b6
ignore-0.4.13 2020-03-22 21:08:37 -04:00
Andrew Gallant
543f99dbf1
grep-regex-0.1.7 2020-03-22 21:08:19 -04:00
Andrew Gallant
0ea65efd6d
regex: special case literal extraction
In a prior commit, we fixed a performance problem with the -w flag by
doing a little extra work to extract literals. It turns out that using
literals in this case when the -w flag is NOT used results in a
performance regression. The reasoning is that we end up using a "fast"
regex as a prefilter when the regex engine itself uses its own
equivalent prefilter, so ripgrep ends up redoing a fair amount of work.

Instead, we only do this extra work when we know the -w flag is enabled.
2020-03-22 21:02:51 -04:00
Andrew Gallant
8ba6ccd159
ignore: fix failing test
This fixes fallout from fixing #1520.
2020-03-16 19:16:24 -04:00
Andrew Gallant
34edb8123a
ignore: squash noisy error message
We should not assume that the commondir file actually exists. If it
doesn't, then just move on. This otherwise emits an error message when
searching normal submodules, which is not OK.

This regression was introduced in #1446.

Fixes #1520
2020-03-16 18:50:02 -04:00
Andrew Gallant
92daa34eb3
ripgrep: release 12.0.0 2020-03-15 21:42:54 -04:00
Andrew Gallant
e772a95b58 regex: avoid using literal optimizations when whitespace is detected
If a literal is entirely whitespace, then it's quite likely that it is
very common. So when that case occurs, just don't do (inner) literal
optimizations at all.

The regex engine may still make sub-optimal decisions here, but that's a
problem for another day.

Fixes #1087
2020-03-15 13:19:14 -04:00
Andrew Gallant
9dd4bf8d7f style: fix rust-analyzer lint warnings 2020-03-15 13:19:14 -04:00
Andrew Gallant
c4c43c733e cli: add --no-ignore-files flag
The purpose of this flag is to force ripgrep to ignore all --ignore-file
flags (whether they come before or after --no-ignore-files).

This flag can be overridden with --ignore-files.

Fixes #1466
2020-03-15 13:19:14 -04:00
Andrew Gallant
447506ebe0 doc: clarify globing behavior
Fixes #1442, Fixes #1478
2020-03-15 13:19:14 -04:00
Andrew Gallant
12e4180985 doc: remove CPU features from man pages
It doesn't really belong in the man page since it's an artifact of a
build/runtime configuration. Moreover, it inhibits reproducible builds.

Fixes #1441
2020-03-15 13:19:14 -04:00
Andrew Gallant
daa8319398 doc: note ripgrep's stdin behavior
Fixes #1439
2020-03-15 13:19:14 -04:00
pierrenn
3a6a24a52a
cli: add engine flag
This permits switching between the different regex engine modes that
ripgrep supports. The purpose of this flag is to make it easier to
extend ripgrep with additional regex engines.

Closes #1488, Closes #1502
2020-03-15 09:30:58 -04:00
pierrenn
aab3d80374
args: refactor to permit adding other engines
This is in preparation for adding a new --engine flag which is intended
to eventually supplant --auto-hybrid-regex.

While there are no immediate plans to add more regex engines to ripgrep,
this is intended to make it easier to maintain a patch to ripgrep with
an additional regex engine. See #1488 for more details.
2020-03-15 09:24:28 -04:00
Andrew Gallant
1856cda77b
style: fix rust-analyzer lints in core 2020-03-15 09:04:54 -04:00
chip
50d2047ae2
crates: update URLs in Cargo.toml
This corrects an oversight when the repo was re-organized to
have its crates moved into a 'crates' sub-directory.

PR #1505
2020-02-28 20:31:43 -05:00
Wolf Honore
227436624f
ignore/types: add coq type
PR #1504
2020-02-28 19:11:29 -05:00
Lucien Greathouse
db7a8cdcb5
globset: Implement serde::{Serialize, Deserialize} for Glob
PR #1492
2020-02-21 07:40:47 -05:00
Andrew Gallant
4176050cdd
ignore: another simplification
Again, thanks to @zsugabubus!
2020-02-20 17:26:34 -05:00
Andrew Gallant
109460fce2
ignore: simplify parallel worker initialization
We can just ask the channel whether any work has been loaded. Normally
querying a channel for its length is a strong predictor of bugs, but in
this case, we do it before we ever attempt a `recv`, so it should work.

Kudos to @zsugabubus for suggesting this!
2020-02-20 16:50:41 -05:00
Andrew Gallant
f314b0d55f ignore: fix parallel traversal
It turns out that the previous version wasn't quite correct. Namely, it
was possible for the following sequence to occur:

1. Consider that all workers, except for one, are `waiting`.
2. The last remaining worker finds one more job to do and sends it on
   the channel.
3. One of the previously `waiting` workers wakes up from the job that
   the last running worker sent, but `self.resume()` has not been
   called yet.
4. The last worker, from (2), calls `get_work` and sees that the
   channel has nothing on it, so it executes `self.waiting() ==
   1`. Since the worker in (3) hasn't called `self.resume()` yet,
   `self.waiting() == 1` evaluates to true.
5. This sets off a chain reaction that stops all workers, despite that
   fact that (3) got more work (which could itself spawn more work).

The end result is that the traversal may terminate while their are still
outstanding work items to process. This problem was observed through
spurious failures in CI. I was not actually able to reproduce the bug
locally.

We fix this by changing our strategy to detect termination using a
counter. Namely, we increment the counter just before sending new work
and decrement the counter just after finishing work. In this way, we
guarantee that the counter only ever reaches 0 once there is no more
work to process.

See #1337 for more discussion. Many thanks to @zsugabubus for helping me
work through this.
2020-02-20 16:07:51 -05:00
asymmetric
b44554c803
ignore/types: add K type
Adds support for files used by the K executable semantic framework:
http://www.kframework.org/index.php/Main_Page

PR #1493
2020-02-19 07:07:09 -05:00
Andrew Gallant
0874aa115c repo: make ripgrep build with the new organization 2020-02-17 19:24:53 -05:00
Andrew Gallant
fdd8510fdd repo: move all source code in crates directory
The top-level listing was just getting a bit too long for my taste. So
put all of the code in one directory and shrink the large top-level mess
to a small top-level mess.

NOTE: This commit only contains renames. The subsequent commit will
actually make ripgrep build again. We do it this way with the naive hope
that this will make it easier for git history to track the renames.
Sigh.
2020-02-17 19:24:53 -05:00