Fix for #1731
Especially on Windows, installing Ruby gems in parallel can fail with a
race condition as `gemspec` files are written simultaneously, and can be
read in a part-written form. This change implements a retry/fallback
strategy to handle this and still keep the majority of the performance
gains of the parallel installation path.
1. Try installing in parallel as before.
2. If there's an error (of any kind), try each gem again after a short,
random delay. The delay is there to try to introduce a bit of jitter, in
case multiple gems all failed and they all then try to install at
exactly the same time again.
3. If that fails too, automatically fall back to sequential install.
This will re-use any gems that were already installed in parallel.
In the happy path case, this has no impact to runtime. This will almost
always be the case on Unix-based systems.
The next path has a slight slow-down due to the retry and the pause, but
it's still pretty quick.
And in the final case, hopefully a fair amount of the work has already
been done.
The drawback of the above, of course, is that it's more complex than the
basic parallel path, and even that's more complex than the basic
sequential process.
The change also includes a tweak to the tests, where the 'waiting for
lock' message occasionally appeared (which may coincide with occasions
when the retry behaviour above was triggered). The warning is now
filtered out and ignored by the tests.
As this is a race condition it's hard to guarantee that this fixes the
problem, but I've run 15 cycles of the windows/ruby test in CI and have
seen no failures.
Current Ruby support can only use Ruby interpreters which are already
installed on the system, although it goes to great lengths to find
interpreters installed by a variety of Ruby managers. This change adds
support for installing new interpreters using the binaries delivered by
the `rv` team. `rv` only provide installers for versions of Ruby still
actively supported (so they don't offer version 3.1, for example), and
only build for a subset of all Ruby-supported platforms. If users need
an unsupported version of Ruby or wish to use an unsupported platform,
they will be prompted to download and install a version of Ruby
manually.
`rv` bundles are named according to the platform, currently including
these components in the filename:
- x86_64_linux
- arm64_linux
- x86_64_linux_musl
- arm64_linux_musl
- ventura (used for macOS on x86_64)
- arm64_sonoma (used for macOS on 64-bit ARM)
If and when upstream `rv` changes these names, the detection code will
need to be updated to match. In particular, this includes the use of
macOS codenames, as if `rv` stop releasing a 'sonoma' package, this will
block installing the macOS versions of Ruby. Currently `rv` seem to be
attempting to keep these codenames, as they already rename their x86_64
builds from 'sequoia' (macOS 15) to 'ventura' (macOS 13). Adding a new
CPU architecture (such as RISC-V) would also need changes, but wouldn't
break existing platform support.
Ruby versions are found by querying the GitHub Releases API, searching
the options returned for an installer that matches the platform and
version requirements, then, if found, downloading and unpacking into the
`prek` tools folder. The `PREK_RUBY_MIRROR` environment variable can be
used to point to a different source for installers, for example to
support mirrors or air-gapped CI environments. Mirrors need to follow
the GitHub URL patterns, but note that although the GitHub hostname
changes between `api.github.com` and `github.com` as needed, any
non-GitHub mirror server will not be remapped in this manner. Where Ruby
is being downloaded from GitHub (either from the upstream `rv` or a
mirror), this remapping does occur, and any `GITHUB_TOKEN` will be sent
with the requests. This both limits impact of rate limiting, and also
allows a private GitHub repository to be used (e.g. for a vetted subset
of `rv` rubies to be mirrored). Note that GitHub tokens will only be
sent to mirrors which are hosted on GitHub.
To allow for passing the `GITHUB_TOKEN` in download requests, the
generic `download_and_extract` function is now a wrapper over a version
which takes an extension function, with the default function not
extending the request. The Ruby code will add the GitHub token if the
request is to GitHub.
Closes#43Closes#765
---------
Co-authored-by: Jo <10510431+j178@users.noreply.github.com>
Setting `pass_filenames: n` limits each hook invocation to at most `n`
filenames, with multiple parallel invocations for larger file sets. This
mirrors the `-n` flag of `xargs` and is useful for tools that can only
process a limited number of files per invocation (typically, if a limit
applies, it is 1).
The existing boolean behaviour is preserved: `true` passes all filenames
(default) and `false` passes none. So this is a backwards compatible
change.
Resolves: #1471
---------
Signed-off-by: JP-Ellis <josh@jpellis.me>
If any rustup toolchain has a broken `rustc` binary (e.g. a
partially-built stage0 sysroot registered via rustup), `prek` would fail
entirely when trying to select a suitable toolchain, even if many valid
toolchains were available.
Closes#1695
## Changes
- **`rustup.rs`**: In both `list_installed_toolchains()` and
`list_system_toolchains()`, replace `try_collect()` with `filter_map()`
so that toolchains whose `rustc --version` fails are skipped with a
`warn!` log rather than propagating the error. The overall operation
succeeds as long as at least one valid toolchain exists.
```rust
// Before: one failure poisons the entire stream
.try_collect()
.await?
// After: bad toolchains are warned and skipped
.filter_map(async move |result| match result {
Ok(info) => Some(info),
Err(e) => {
warn!("Skipping invalid toolchain: {e:#}");
None
}
})
.collect()
.await
```
<!-- START COPILOT ORIGINAL PROMPT -->
<details>
<summary>Original prompt</summary>
----
*This section details on the original issue you should resolve*
<issue_title>One invalid Rust toolchain causes prek to
fail</issue_title>
<issue_description>### Summary
When running a Rust-based hook, I've got the following error:
```
error: Failed to install hook `oxipng`
caused by: Failed to install rust
caused by: Failed to read version from /home/m4tx/projects/rust/build/host/stage0-sysroot/bin/rustc
caused by: Run command `rustc version` failed
caused by: No such file or directory (os error 2)
```
This was very concerning for me at first, since `/home/m4tx/projects` is
a very arbitrary path that I happen to store various repositories in,
and I would never expect prek to look for anything in there. Running
prek with `-vvv` did shine some light on this:
```
2026-02-26T16:45:23.963699Z TRACE Executing `/home/m4tx/.rustup/toolchains/nightly-2025-07-03-x86_64-unknown-linux-gnu/bin/rustc --version`
2026-02-26T16:45:23.963942Z TRACE Executing `/home/m4tx/.rustup/toolchains/1.42.0-x86_64-unknown-linux-gnu/bin/rustc --version`
2026-02-26T16:45:23.964608Z TRACE Executing `/home/m4tx/.rustup/toolchains/1.57.0-x86_64-unknown-linux-gnu/bin/rustc --version`
2026-02-26T16:45:23.964848Z TRACE Executing `/home/m4tx/.rustup/toolchains/1.60-x86_64-unknown-linux-gnu/bin/rustc --version`
2026-02-26T16:45:23.965053Z TRACE Executing `/home/m4tx/.rustup/toolchains/1.64.0-x86_64-unknown-linux-gnu/bin/rustc --version`
2026-02-26T16:45:23.965181Z TRACE Executing `/home/m4tx/.rustup/toolchains/1.70.0-x86_64-unknown-linux-gnu/bin/rustc --version`
2026-02-26T16:45:23.965471Z TRACE Executing `/home/m4tx/.rustup/toolchains/1.75.0-x86_64-unknown-linux-gnu/bin/rustc --version`
2026-02-26T16:45:23.966651Z TRACE Executing `/home/m4tx/.rustup/toolchains/1.77.2-x86_64-unknown-linux-gnu/bin/rustc --version`
2026-02-26T16:45:23.972872Z TRACE Executing `/home/m4tx/.rustup/toolchains/1.78.0-x86_64-unknown-linux-gnu/bin/rustc --version`
2026-02-26T16:45:23.975137Z TRACE Executing `/home/m4tx/.rustup/toolchains/1.79.0-x86_64-unknown-linux-gnu/bin/rustc --version`
2026-02-26T16:45:23.977357Z TRACE Executing `/home/m4tx/.rustup/toolchains/1.80.1-x86_64-unknown-linux-gnu/bin/rustc --version`
2026-02-26T16:45:23.977749Z TRACE Executing `/home/m4tx/.rustup/toolchains/1.81.0-x86_64-unknown-linux-gnu/bin/rustc --version`
2026-02-26T16:45:23.977995Z TRACE Executing `/home/m4tx/.rustup/toolchains/1.84.1-x86_64-unknown-linux-gnu/bin/rustc --version`
2026-02-26T16:45:23.978252Z TRACE Executing `/home/m4tx/.rustup/toolchains/1.85-x86_64-unknown-linux-gnu/bin/rustc --version`
2026-02-26T16:45:23.978423Z TRACE Executing `/home/m4tx/.rustup/toolchains/1.85.1-x86_64-unknown-linux-gnu/bin/rustc --version`
2026-02-26T16:45:23.978681Z TRACE Executing `/home/m4tx/.rustup/toolchains/1.93-x86_64-unknown-linux-gnu/bin/rustc --version`
2026-02-26T16:45:23.983378Z TRACE Executing `/home/m4tx/projects/rust/build/host/stage0-sysroot/bin/rustc --version`
```
Turns out that I was trying to compile Rust from scratch, which resulted
in the toolchain being added to `rustup`. For the next 6 months ran into
any issues because of this (since I had a number of other perfectly
working toolchains installed) until I tried prek. Removing the
problematic toolchain from rustup has fixed the problem.
I don't think that `prek` should fail on _any_ when it encounters
failure using any toolchain; rather, I would expect it to work if at
least one Rust toolchain is usable.
### Willing to submit a PR?
- [x] Yes — I’m willing to open a PR to fix this.
### Platform
Arch Linux, Linux 6.12.74-1-lts x86_64 GNU/Linux
### Version
prek 0.3.3
### .pre-commit-config.yaml
```yaml
repos:
- repo: https://github.com/oxipng/oxipng
rev: v9.1.4
hooks:
- id: oxipng
args: ["-o", "max", "--strip", "safe", "--alpha"]
```
### Log file
```
2026-02-26T16:45:23.937725Z DEBUG prek: 0.3.3
2026-02-26T16:45:23.937755Z DEBUG Args: ["prek", "-vvv"]
2026-02-26T16:45:23.938885Z TRACE get_root: close time.busy=1.10ms time.idle=2.87µs
2026-02-26T16:45:23.938914Z DEBUG Git root: /home/m4tx/projects/m4txblog
2026-02-26T16:45:23.938928Z TRACE Executing `/usr/bin/git ls-files --unmerged`
2026-02-26T16:45:23.939978Z DEBUG Found workspace root at `/home/m4tx/projects/m4txblog`
2026-02-26T16:45:23.939988Z TRACE Include selectors: ``
2026-02-26T16:45:23.939993Z TRACE Skip selectors: ``
2026-02-26T16:45:23.940045Z DEBUG discover{root="/home/m4tx/projects/m4txblog" config=None refresh=false}: Loaded workspace from cache
2026-02-26T16:45:23.940061Z DEBUG discover{root="/home/m4tx/projects/m4txblog" config=None refresh=false}: Loading project configuration path=.pre-commit-config.yaml
2026-02-26T16:45:23.940757Z TRACE discover{root="/home/m4tx/projects/m4txblog" config=None refresh=false}: close time.busy=748µs time.idle=1.18µs
2026-02-26T16:45:23.940780Z TRACE Executing `/usr/bin/git diff --exit-code --name-only -z /home/m4tx/projects/m4txblog/.pre-commit-config.yaml`
2026-02-26T1...
</details>
<!-- START COPILOT CODING AGENT SUFFIX -->
- Fixesj178/prek#1695
<!-- START COPILOT CODING AGENT TIPS -->
---
🔒 GitHub Advanced Security automatically protects Copilot coding agent pull requests. You can protect all pull requests by enabling Advanced Security for your repositories. [Learn more about Advanced Security.](https://gh.io/cca-advanced-security)
---------
Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: j178 <10510431+j178@users.noreply.github.com>
Related Issue: https://github.com/j178/prek/issues/1685
## Summary
prek can now be run with python -m prek in addition to the prek command,
matching common Python CLI tools like `black`, `ruff`, and `uv`.
## Motivation
- Consistency: Many Python tools support python -m <package>
- Virtual environments: Makes it clear which Python interpreter (and
venv) is used
- Scripts & CI: Explicit execution environment
- Tool integration: Some tools only support the `python -m` form
## Changes
- Add `python/prek/` package with `__init__.py` and `__main__.py`
- Set python-source = "python" in `pyproject.toml` so maturin includes
the Python package
- `__main__.`py invokes the prek binary next to the Python executable
(e.g. `.venv/bin/prek with .venv/bin/python`), falling back to PATH if
not found
---------
Co-authored-by: Jo <10510431+j178@users.noreply.github.com>
## Summary
- Add `PREK_MAX_CONCURRENCY` environment variable to cap the maximum
number of concurrent hooks
- Value is floored at 1; values above CPU count are allowed (useful for
I/O-bound hooks)
- Invalid values show a user-visible warning and fall back to CPU count
- `PREK_NO_CONCURRENCY` takes precedence when both are set
## Motivation
When `ulimit -n` is low, concurrent hook execution can exhaust file
descriptors. This provides an environment variable to limit concurrency.
For #1696
## Test plan
- [x] `mise run lint` — no warnings
- [x] `mise run test` — all tests pass
- [x] Unit tests for `resolve_concurrency`: valid value, above CPU
count, zero floors to 1, invalid string, empty string, unset,
no_concurrency, no_concurrency overrides max
Co-authored-by: Claude <noreply@anthropic.com>
Currently the gem support code runs a single `gem install` to install
the gems in the hook. Dependencies are resolved by the `gem` command,
which installs each dependency in sequence. Any gems with native code
are compiled, with the compilation stage of each gem using only a single
compile task.
With this change, gems are installed in parallel (both top-level and
dependencies), and any gems with native code allow these gems to use
multiple compiler tasks (so using multiple cores where available). (Note
that running multiple compilers will be the default `gem` behaviour in
Ruby 4, so this ought to be safe).
The benefits of this will vary according to computer and hook, but an
approximately 3x speed increase has been observed with the rubocop hook
(from 38s down to 12s).
Additional speedup for https://github.com/j178/prek/issues/43
---------
Co-authored-by: Jo <10510431+j178@users.noreply.github.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Trace-level `Executing ...` lines currently truncate command arguments
at a hardcoded 120 characters, which hides critical details when
debugging hooks (notably `docker_image` entrypoint/args). This change
introduces a configurable truncate limit while preserving existing
default behavior.
- **Config surface: new env var**
- Added `PREK_LOG_TRUNCATE_LIMIT` to `prek-consts` (`EnvVars`).
- **Runtime behavior in command logging**
- Updated `Cmd` display formatting in `crates/prek/src/process.rs` to
read truncation limit from `PREK_LOG_TRUNCATE_LIMIT`.
- Cached the resolved truncate limit with `LazyLock` so env parsing
happens once.
- Kept default at `120` when unset/invalid.
- Treated `0` as invalid and fallback to default to avoid degenerate
output.
- **Documentation**
- Added `PREK_LOG_TRUNCATE_LIMIT` to `docs/configuration.md` under
environment variables, including default and intent.
- **Focused unit coverage**
- Added tests for:
- env var value parsing (valid/invalid/zero),
- env-based resolution behavior,
- cached-limit behavior via `LazyLock`,
- env restoration guard to avoid cross-test env leakage.
```rust
const DEFAULT_COMMAND_LOG_TRUNCATE_LIMIT: usize = 120;
static COMMAND_LOG_TRUNCATE_LIMIT: LazyLock<usize> =
LazyLock::new(command_log_truncate_limit_from_env);
fn command_log_truncate_limit() -> usize {
*COMMAND_LOG_TRUNCATE_LIMIT
}
```
<!-- START COPILOT ORIGINAL PROMPT -->
<details>
<summary>Original prompt</summary>
>
> ----
>
> *This section details on the original issue you should resolve*
>
> <issue_title>Do not truncate trace run executions</issue_title>
> <issue_description>### Summary
>
> I was trying to debug a docker_image hook execution and was delighted
to note the --log-file option.
>
> However, turns out that the "Executing" TRACE run lines are truncated
with `[...]` before showing the part I'm actually interested in (a case
of tweaking entrypoints and arguments).
>
> I'd suggest not truncating these lines in the trace log at all, or
providing an option not to truncate.
>
> ### Willing to submit a PR?
>
> - [ ] Yes — I’m willing to open a PR to fix this.
>
> ### Platform
>
> Ubuntu 25.10 amd64
>
> ### Version
>
> 0.3.2
>
> ### .pre-commit-config.yaml
>
> N/A
>
> ### Log file
>
> ```
> 2026-02-20T10:17:47.223740Z TRACE run{hook_id=...
language=docker_image}: Executing `cd ... && docker run --rm --tty
--user 1000:1000 --init --volume ...:/src:rw,Z --workdir /src
--entrypoint ... --network [...]`
> ```</issue_description>
>
> <agent_instructions>Add a new env var to prek-consts, controlling the
command line log truncate limit in process.rs, current it is hardcoded
to 120. Document it in docs/configuration.md env vars
section.</agent_instructions>
>
> ## Comments on the Issue (you are @copilot in this section)
>
> <comments>
> </comments>
>
</details>
<!-- START COPILOT CODING AGENT SUFFIX -->
- Fixesj178/prek#1678
<!-- START COPILOT CODING AGENT TIPS -->
---
🔒 GitHub Advanced Security automatically protects Copilot coding agent
pull requests. You can protect all pull requests by enabling Advanced
Security for your repositories. [Learn more about Advanced
Security.](https://gh.io/cca-advanced-security)
---------
Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: j178 <10510431+j178@users.noreply.github.com>
