1
0
mirror of https://github.com/facebook/zstd.git synced 2025-03-06 08:49:28 +02:00

4713 Commits

Author SHA1 Message Date
Po-Chuan Hsieh
103a85e6f6
Use md5sum rather than gmd5sum for FreeBSD
Reference:	https://man.freebsd.org/cgi/man.cgi?query=md5sum
2024-03-27 20:53:52 +08:00
Yann Collet
c6e5257240
Merge pull request #3977 from facebook/doc_advanced
Doc update
2024-03-21 12:33:15 -07:00
Nick Terrell
731f4b70fc Fix & fuzz ZSTD_generateSequences
This function was seriously flawed:
* It didn't do output bounds checks
* It produced invalid sequences when an uncompressed or RLE block was emitted
* It produced invalid sequences when the block splitter was enabled
* It produced invalid sequences when ZSTD_c_targetCBlockSize was enabled

I've attempted to fix these issues, but this function is just a bad idea,
so I've marked it as deprecated and unsafe. We should replace it with
`ZSTD_extractSequences()` which operates on a compressed frame.
2024-03-21 07:18:05 -07:00
Elliot Gorokhovsky
741b87bbe1
Fuzzing and bugfixes for magicless-format decoding (#3976)
* fuzzing and bugfixes for magicless format

* reset dctx before each decompression

* do not memcmp empty buffers

* nit: decompressor errata
2024-03-20 19:22:34 -04:00
Yann Collet
6f1215b874 fix ZSTD_TARGETCBLOCKSIZE_MIN test
when requested CBlockSize is too low,
bound it to the minimum
instead of returning an error.
2024-03-18 14:10:08 -07:00
Yann Collet
c5da438dc0 fix typo 2024-03-18 12:33:22 -07:00
Yann Collet
902c7ec1fe add doc on CCtx UB state 2024-03-18 12:30:35 -07:00
Yann Collet
5d82c2b57c add a paragraph on UB DCtx state after error 2024-03-18 12:17:41 -07:00
Yann Collet
f5728da365 update targetCBlockSize documentation 2024-03-18 12:04:02 -07:00
Elliot Gorokhovsky
7d970bd83c
Implement one-shot fallback for magicless format (#3971) 2024-03-18 10:55:53 -04:00
Yann Collet
0fcdc62c2d
Merge pull request #3969 from facebook/v156_prep
bump version number
2024-03-15 10:26:49 -07:00
Yann Collet
dff5407cd0
Merge pull request #3966 from facebook/debug_lineNumber
add line number to debug traces
2024-03-15 10:26:06 -07:00
Yann Collet
686e7e4b4b updated version to v1.5.6 2024-03-14 15:38:14 -07:00
Elliot Gorokhovsky
559762da12
Remove duplicate and incorrect docs in zstd_decompress.c (#3967) 2024-03-14 15:55:01 -04:00
Yann Collet
9cc3304614 add line number to debug traces 2024-03-14 12:11:11 -07:00
Felix Handte
515c07a131
Merge pull request #3964 from felixhandte/promote-tgt-c-blk-size-to-stable
Promote `ZSTD_c_targetCBlockSize` Parameter to Stable API
2024-03-14 13:06:46 -04:00
Yann Collet
216099a73f
Merge pull request #3962 from facebook/cover_lessIncludes
reduce the amount of #include in cover.h
2024-03-13 16:52:54 -07:00
W. Felix Handte
3613448fb8 Promote ZSTD_c_targetCBlockSize Parameter to Stable API
This feature has demonstrated itself to be useful in web compression and we
want to encourage other folks to use it. But we currently make it difficult
to do so since it's locked away in the experimental API.

The API itself is really straightforward and I think it's fine to commit to
maintaining support / compatibility for this API even if in the future the
underlying implementation may continue to evolve.

Note that this commit changes its enum name and also its numeric value. Users
who respected the instructions of using the experimental API should be fine
with both of these changes since they should only have referred to it by the.

Conceivably someone could have done bad feature detection of this capability
by doing `#ifdef ZSTD_c_targetCBlockSize` which will now return false since
it's no longer a macro... but I think that's an acceptable hypothetical
breakage.
2024-03-13 17:07:10 -04:00
Nick Terrell
ff0afbad58 [asm][aarch64] Mark that BTI and PAC are supported
Mark that `huf_decompress_amd64.S` supports BTI and PAC, which it trivially does because it is empty for aarch64.

The issue only requested BTI markings, but it also makes sense to mark PAC, which is the only other feature.

Also run add a test for this mode to the ARM64 QEMU test. Before this PR it warns on `huf_decompress_amd64.S`, after it doesn't.

Fixes Issue #3841.
2024-03-13 16:15:51 -04:00
Yann Collet
c8ab027227 reduce the amount of includes in "cover.h" 2024-03-13 11:29:28 -07:00
Nick Terrell
94c102038b [cpu] Backport fix for rbx clobbering on Windows with Clang
Backport folly fix for rbx clobbering: f22f88b8b9

This supercedes PR #3646.
2024-03-13 09:45:40 -04:00
Yonatan Komornik
b20703f273
Updates ZSTD_RowFindBestMatch comment (#3947)
Updates the comment on the head of `ZSTD_RowFindBestMatch` to make sure it's aligned with recent changes to the hash table.
2024-03-12 15:10:07 -07:00
Elliot Gorokhovsky
f65b9e27ce
Exercise ZSTD_findDecompressedSize() in the simple decompression fuzzer (#3959)
* Improve decompression fuzzer

* Fix legacy frame header fuzzer crash, add unit test
2024-03-12 17:07:06 -04:00
Yann Collet
db996d253e
Merge pull request #3933 from facebook/fix3819
prevent XXH64 from being autovectorized by XXH512 by default
2024-03-12 09:46:48 -07:00
Yann Collet
3b573ba1f0
Merge pull request #3936 from facebook/debug5
minor: fix incorrect debug level
2024-03-12 09:45:20 -07:00
Yann Collet
fe7c645a7e
Merge pull request #3937 from facebook/offset0
new method to deal with offset==0 erroneous edge case
2024-03-12 09:25:06 -07:00
acceptacross
8ba5bc4729
chore: fix some typos (#3949)
Signed-off-by: acceptacross <csqcqs@gmail.com>
2024-03-12 11:44:42 -04:00
Elliot Gorokhovsky
e0872806df
Use ZSTD_LEGACY_SUPPORT=5 in make test (#3943) 2024-03-12 10:08:26 -04:00
Yann Collet
372fddf4e6
Merge pull request #3860 from likema/fix-xxhash-aix-51
Fix building xxhash on AIX 5.1
2024-03-09 15:34:02 -08:00
Yann Collet
a9fb8d4c41 new method to deal with offset==0
in this new method, when an `offset==0` is detected,
it's converted into (size_t)(-1), instead of 1.

The logic is that (size_t)(-1) is effectively an extremely large positive number,
which will not pass the offset distance test at next stage (`execSequence()`).
Checked the source code, and offset is always checked (as it should),
using a formula which is not vulnerable to arithmetic overflow:
```
RETURN_ERROR_IF(sequence.offset > (size_t)(oLitEnd - virtualStart),
```

The benefit is that such a case (offset==0) is always detected as corrupted data
as opposed to relying on the checksum to detect the error.
2024-03-08 15:26:06 -08:00
Yann Collet
aed172a8fe minor: fix incorrect debug level 2024-03-08 14:29:44 -08:00
Yann Collet
007cda88ca prevent XXH64 from being autovectorized by XXH512 by default
backport fix https://github.com/Cyan4973/xxHash/pull/924 from libxxhash
2024-03-07 16:43:13 -08:00
Yann Collet
8689633fdf
Merge pull request #3840 from aimuz/fix-reserved
lib/decompress: check for reserved bit corruption in zstd
2024-03-05 13:40:12 -08:00
Yann Collet
e385c3dd46
Merge pull request #3753 from facebook/make2
minor Makefile refactoring
2024-03-03 19:13:00 -08:00
Yann Collet
8d31e8ec42 sizeBlockSequences() also tracks uncompressed size
and only defines a sub-block boundary when
it believes that it is compressible.

It's effectively an optimization,
avoiding a compression cycle to reach the same conclusion.
2024-02-26 14:31:12 -08:00
Yann Collet
d23b95d21d minor refactor for clarity
since we can ensure that nbSubBlocks>0
2024-02-26 14:06:34 -08:00
Yann Collet
86db60752d optimization: bail out faster in presence of incompressible data 2024-02-26 13:27:59 -08:00
Yann Collet
ef82b214ad nit: comment indentation
as reported by @terrelln
2024-02-26 13:23:59 -08:00
Yann Collet
aa8592c532 minor: reformulate nbSubBlocks assignment 2024-02-26 13:21:14 -08:00
Yann Collet
e0412c2062 fix extraneous semicolon ';'
as reported by @terrelln
2024-02-26 12:26:54 -08:00
Yann Collet
1fafd0c4ae fix minor visual static analyzer warning
it's a false positive,
but change the code nonetheless to make it more obvious to the static analyzer.
2024-02-25 19:45:32 -08:00
Yann Collet
038a8a906b targetCBlockSize: modified splitting strategy to generate blocks of more regular size
notably avoiding to feature a larger first block
2024-02-25 17:39:29 -08:00
Yann Collet
f8372191f5 reduced minimum compressed block size
with the intention to match the transport layer size,
such as Ethernet and 4G mobile networks.
2024-02-24 01:59:16 -08:00
Yann Collet
f77f634d41 update API documentation 2024-02-24 01:28:17 -08:00
Yann Collet
4b51526412 fix partial block uncompressed 2024-02-24 01:24:58 -08:00
Yann Collet
6719794379 fixed some regressionTests
but not all
2024-02-23 18:48:29 -08:00
Yann Collet
0591e7eea1 minor: fix overly cautious conversion warning 2024-02-23 16:05:09 -08:00
Yann Collet
3b40100058 fix long sequences (> 64 KB) 2024-02-23 15:35:12 -08:00
Yann Collet
6b11fc436c fix issue with incompressible sections 2024-02-23 14:53:56 -08:00
Yann Collet
cc4530924b speed optimized version of targetCBlockSize
note that the size of individual compressed blocks will vary more wildly with this modification.
But it seems good enough for a first test, and fix the speed regression issue.
Further refinements can be attempted later.
2024-02-23 14:03:26 -08:00