Victor Zhang
6b046f5841
PR feedback
2025-01-02 15:05:58 -08:00
Victor Zhang
757e29e170
Oops
2025-01-02 14:17:24 -08:00
Victor Zhang
54c3d998a0
Support for libc variants without fseeko/ftello
...
Some older Android libc implementations don't support `fseeko` or `ftello`.
This commit adds a new compile-time macro `LIBC_NO_FSEEKO` as well as a usage in CMake for old Android APIs.
2025-01-02 14:02:10 -08:00
Yann Collet
f5d9d57fe6
Merge pull request #4228 from facebook/skippableMagicVariant
...
ZSTD_getFrameHeader() for skippable frames
2025-01-02 10:57:49 -08:00
Yann Collet
04a2a0219c
update type names
...
naming convention: Type names should start with a Capital letter (after the prefix)
2024-12-29 14:25:33 -08:00
Yann Collet
cf5e53f618
Merge pull request #4227 from facebook/doc_skippableFrame
...
clarify documentation for Frame-level methods invoked on a skippable frame
2024-12-29 13:27:35 -08:00
Yann Collet
a2ff6ea784
improve ZSTD_getFrameHeader on skippable frames
...
now reports:
- the header size
- the magic variant (within @dictID field)
2024-12-29 12:26:04 -08:00
Yann Collet
f8a2b352d6
clarify doc on Frame-level methods when invoked on a skippable frame
...
following discussion at #4226
2024-12-29 02:41:09 -08:00
Yann Collet
788926fe48
Merge pull request #4217 from facebook/ZSTD_compressSequencesAndLiterals
...
ZSTD_compressSequencesAndLiterals
2024-12-26 11:41:44 -08:00
Yann Collet
72ce56b527
fixed another invalid scenario
...
compressSequencesAndLiterals() doesn't support sequence validation
2024-12-23 21:15:50 -08:00
Yann Collet
7b294caf46
add one valid test case
...
ZSTD_compressSequencesAndLiterals() may return a specific error code
when data to compress is non-compressible.
2024-12-23 19:43:17 -08:00
Yann Collet
f8725e80cc
added fuzzer test for compressSequencesAndLiterals()
...
piggy-backing onto existing compressSequences() fuzzer test
2024-12-23 18:42:51 -08:00
Nick Terrell
f0937b83d9
[cmake] Fix -z noexecstack portability
...
Summary:
Issue reported by @ryandesign and @MarcusCalhoun-Lopez.
CMake doesn't support spaces in flags. This caused older versions of gcc to
ignore the unknown flag "-z noexecstack" on MacOS since it was interpreted as a
single flag, not two separate flags. Then, during compilation it was treated as
"-z" "noexecstack", which was correctly forwarded to the linker. But the MacOS
linker does not support `-z noexecstack` so compilation failed.
The fix is to use `-Wl,-z,noexecstack`. This is never misinterpreted by a
compiler. However, not all compilers support this syntax to forward flags to the
linker. To fix this issue, we check if all the relevant `noexecstack` flags have
been successfully set, and if they haven't we disable assembly.
See also PR#4056 and PR#4061. I decided to go a different route because this is
simpler. It might not successfully set these flags on some compilers, but in that
case it also disables assembly, so they aren't required.
Test Plan:
```
mkdir build-cmake
cmake ../build/cmake/CMakeLists.txt
make -j
```
See that the linker flag is successfully detected & that assembly is enabled.
Run the same commands on MacOS which doesn't support `-Wl,-z,noexecstack` and see
that everything compiles and that `LD_FLAG_WL_Z_NOEXECSTACK` and
`ZSTD_HAS_NOEXECSTACK` are both false.
2024-12-20 15:06:23 -08:00
Yann Collet
47cbfc87a9
restore invocation of ZSTD_entropyCompressSeqStore()
...
in the ZSTD_compressSequences() pipeline
2024-12-20 10:37:01 -08:00
Yann Collet
522adc34eb
minor: use MEM_writeLE24()
...
so that an empty frame needs only 3 bytes of dstCapacity.
2024-12-20 10:37:01 -08:00
Yann Collet
b7a9e69d8d
added parameter litCapacity
...
to ZSTD_compressSequencesAndLiterals()
to enforce the litCapacity >= litSize+8 condition.
2024-12-20 10:37:01 -08:00
Yann Collet
76445bb379
add a check, to return an error if Sequence validation is enabled
...
since ZSTD_compressSequencesAndLiterals() doesn't support it.
2024-12-20 10:37:01 -08:00
Yann Collet
52a9bc6fca
fixed minor error in preparation of one fullbench scenario
2024-12-20 10:37:00 -08:00
Yann Collet
ab0f1798e8
ensure that srcSize is controlled
2024-12-20 10:37:00 -08:00
Yann Collet
b339efff2b
add dedicated error code for special case
...
ZSTD_compressSequencesAndLiterals() cannot produce an uncompressed block
2024-12-20 10:37:00 -08:00
Yann Collet
a80f55f47d
added a test for ZSTD_compressSequencesAndLiterals
...
checks that srcSize is present in the frame header
and bounds the window size.
2024-12-20 10:37:00 -08:00
Yann Collet
0a54f6f288
ZSTD_compressSequencesAndLiterals requires srcSize as parameter
...
this makes it possible to adjust windowSize to its tightest.
2024-12-20 10:37:00 -08:00
Yann Collet
ad023b392f
fixed minor error in one benchmark scenario
2024-12-20 10:37:00 -08:00
Yann Collet
b7b4e86347
fixed minor conversion warning
2024-12-20 10:37:00 -08:00
Yann Collet
12c47d3262
improved speed of the Sequences converter
2024-12-20 10:37:00 -08:00
Yann Collet
95ad9e47ff
added benchmark for ZSTD_convertBlockSequences_wBlockDelim()
2024-12-20 10:37:00 -08:00
Yann Collet
d48e330ae1
change name to ZSTD_convertSequences*()
2024-12-20 10:37:00 -08:00
Yann Collet
61ac8311e0
attempt to silence Visual Studio warning about fopen()
2024-12-20 10:37:00 -08:00
Yann Collet
f617e86b71
fixed incorrect assert
2024-12-20 10:37:00 -08:00
Yann Collet
47edd0acf4
removed fullbench-dll project from visual solutions
2024-12-20 10:37:00 -08:00
Yann Collet
6f8c1046d0
update Visual Studio solutions
2024-12-20 10:37:00 -08:00
Yann Collet
f0d0d95234
added tests
...
check that ZSTD_compressAndLiterals() also controls that the `srcSize` field is exact.
2024-12-20 10:37:00 -08:00
Yann Collet
31b5ef2539
ZSTD_compressSequencesAndLiterals() now supports multi-blocks frames.
2024-12-20 10:36:59 -08:00
Yann Collet
5164d44dab
change advanced parameter name: ZSTD_c_repcodeResolution
...
and updated its documentation.
Note: older name ZSTD_c_searchForExternalRepcodes remains supported via #define
2024-12-20 10:36:59 -08:00
Yann Collet
ca8bd83373
minor: cleaner function parameter repcodeResolution
2024-12-20 10:36:59 -08:00
Yann Collet
d2d0fdac42
updated documentation on validateSequence
2024-12-20 10:36:59 -08:00
Yann Collet
1f6d6815c3
optimization: instantiate specialized version without Sequence checking code
...
results in +4% compression speed,
thanks to removal of branches in the hot loop.
2024-12-20 10:36:59 -08:00
Yann Collet
a288751de7
minor optimization: only track seqPos->posInSrc when validateSequences is enabled
...
note: very minor saving, no performance impact
2024-12-20 10:36:59 -08:00
Yann Collet
f176514467
minor doc update
2024-12-20 10:36:59 -08:00
Yann Collet
1c8f5b0f11
minor optimization for ZSTD_compressSequencesAndLiterals()
...
does not need to track and update internal `litPtr`.
note: does not measurably impact performance.
2024-12-20 10:36:59 -08:00
Yann Collet
0a5c0807af
minor conversion warning fix
2024-12-20 10:36:59 -08:00
Yann Collet
f281497aef
fullbench: new scenario: compressSequencesAndLiterals()
2024-12-20 10:36:59 -08:00
Yann Collet
ac05ea89a5
fullbench: switch default generator to lorem ipsum
...
which creates more "realistic" scenarios than former compressible noise.
The legacy data generator remains accessible,
it is triggered when requesting an explicit compressibility factor (-P#).
2024-12-20 10:36:59 -08:00
Yann Collet
8ab04097ed
add the compressSequences() benchmark scenario
2024-12-20 10:36:59 -08:00
Yann Collet
c050ae4fb8
fullbench: preparation functions are now in charge of allocating buffers
2024-12-20 10:36:59 -08:00
Yann Collet
8b7e1b795d
minor variable renaming
2024-12-20 10:36:59 -08:00
Yann Collet
c540976a4b
minor: test reordering
2024-12-20 10:36:59 -08:00
Yann Collet
4c097b4939
refactor fullbench
...
to make it easier to add new scenarios
2024-12-20 10:36:58 -08:00
Yann Collet
09964c6276
prepare fullbench for a refactor
2024-12-20 10:36:58 -08:00
Yann Collet
0b013b2688
added unit tests to ZSTD_compressSequencesAndLiterals()
...
seems to work as expected,
correctly control that `litSize` and `srcSize` are exactly correct.
2024-12-20 10:36:58 -08:00
