Align the declarations of BIT_readBits() and BIT_reloadDStream() in
bitstream.h with their FORCE_INLINE_TEMPLATE definitions.
The previous MEM_STATIC declarations caused an attribute mismatch
between the header and the definitions, which can lead to incorrect
compiler assumptions under certain toolchains and optimization levels.
Signed-off-by: Alexander Moch <mail@alexmoch.com>
When pthread_mutex_init() or pthread_cond_init() fails in the debug
implementation (DEBUGLEVEL >= 1), the previously allocated memory was
not freed, causing a memory leak.
This fix ensures that allocated memory is properly freed when pthread
initialization functions fail, preventing resource leaks in error
conditions.
The issue affects:
- ZSTD_pthread_mutex_init() at lib/common/threading.c:146
- ZSTD_pthread_cond_init() at lib/common/threading.c:167
This is particularly important for long-running applications or
scenarios with resource constraints where pthread initialization
might fail due to system limits.
The existing scalar implementation uses a 4-way pipelined histogram
calculation which is very efficient on out-of-order CPUs. However,
this can be further accelerated using the SVE2 HISTSEG instructions -
which compute a histogram for 16 byte chunks in a vector register.
On a system with 128-bit vectors (VL128) we need 16 HISTSEG executions
to compute the histogram for the whole symbol space (0..255) of 16
bytes input. However we can only accumulate 15 of such 16 byte strips
before possible overflow. So we need to extend and save the 8-bit
histogram accumulators to 16-bit after every 240 byte chunks of input.
To store all in registers we would need 32 128-bit registers. Longer
SVE2 vectors could help here, if such machines become available.
The maximum input block size in Zstd is 128 KiB, so 16-bit accumulators
would not be enough. However an LZ pass will prepend the histogram
calculation, so it is impossible (my assumption) to overflow the 16-bit
accumulators.
The symbol distribution is also not uniform, the lower values are more
common, so we used a 3 pass algorithm to prevent stack spilling. In the
first pass we only compute histograms for 64 symbols (4-way SIMD) while
also computing the maximum symbol value. If we have symbol values
larger than 64 we start the second pass to compute the next 96 elements
of the histogram. The final pass calculates the remaining part of the
histogram (256 symbols in total) if needed. This split of histogram
generation gave the best overall results for performance.
This implementation is the best performing of a number of different
cache blocking schemes tested.
Compression uplifts on a Neoverse V2 system, using Zstd-1.5.8
(e26dde3d) as a baseline, compiled with "-O3 -march=armv8.2-a+sve2":
Clang-20 GCC-14
1#silesia.tar: +6.173% +5.987%
2#silesia.tar: +5.200% +5.011%
3#silesia.tar: +4.332% +5.031%
4#silesia.tar: +2.789% +3.064%
5#silesia.tar: +2.028% +1.838%
6#silesia.tar: +1.562% +1.340%
7#silesia.tar: +1.160% +0.959%
technically equivalent to `size_t`,
but it's the proper type for underlying register representation.
This makes it possible to control register type, and therefore size, independently from `size_t`,
which can be useful on systems where `size_t` is 32-bit, while the architecture supports 64-bit registers.
this was previously no triggered in x86 32-bit mode,
due to a limitation in `bitstream.h`, that was fixed in #4248.
Now, `bmi2` will be automatically detected and triggered
at compilation time, if the corresponding instruction set is enabled,
even in 32-bit mode.
Also: updated library documentation, to feature STATIC_BMI2 build variable
NiDU-NINJA
Sergey G. Brester (sebres)
Alexander Moch
ZijianLi
Ryan Lefkowitz
Arpad Panyik
Nick Terrell
Yann Collet
Yann Collet
Pavel P
Yann Collet
MessyHack
Victor Zhang