Elliot Gorokhovsky
ff42ed1582
Rename "External Matchfinder" to "Block-Level Sequence Producer" ( #3484 )
...
* change "external matchfinder" to "external sequence producer"
* migrate contrib/ to new naming convention
* fix contrib build
* fix error message
* update debug strings
* fix def of invalid sequences in zstd.h
* nit
* update CHANGELOG
* fix .gitignore
2023-02-09 17:01:17 -05:00
Yann Collet
db18a62f89
Provide more accurate error codes for busy-loop scenarios
...
fixes #3454
2023-01-25 13:07:53 -08:00
Danielle Rozenblit
815d1d4eda
update external sequence error to fit error naming scheme
2023-01-23 09:58:34 -08:00
Danielle Rozenblit
1b65727e74
fix nits and add new error code for invalid external sequences
2023-01-23 07:59:02 -08:00
Elliot Gorokhovsky
2a402626dd
External matchfinder API ( #3333 )
...
* First building commit with sample matchfinder
* Set up ZSTD_externalMatchCtx struct
* move seqBuffer to ZSTD_Sequence*
* support non-contiguous dictionary
* clean up parens
* add clearExternalMatchfinder, handle allocation errors
* Add useExternalMatchfinder cParam
* validate useExternalMatchfinder cParam
* Disable LDM + external matchfinder
* Check for static CCtx
* Validate mState and mStateDestructor
* Improve LDM check to cover both branches
* Error API with optional fallback
* handle RLE properly for external matchfinder
* nit
* Move to a CDict-like model for resource ownership
* Add hidden useExternalMatchfinder bool to CCtx_params_s
* Eliminate malloc, move to cwksp allocation
* Handle CCtx reset properly
* Ensure seqStore has enough space for external sequences
* fix capitalization
* Add DEBUGLOG statements
* Add compressionLevel param to matchfinder API
* fix c99 issues and add a param combination error code
* nits
* Test external matchfinder API
* C90 compat for simpleExternalMatchFinder
* Fix some @nocommits and an ASAN bug
* nit
* nit
* nits
* forward declare copySequencesToSeqStore functions in zstd_compress_internal.h
* nit
* nit
* nits
* Update copyright headers
* Fix CMake zstreamtest build
* Fix copyright headers (again)
* typo
* Add externalMatchfinder demo program to make contrib
* Reduce memory consumption for small blockSize
* ZSTD_postProcessExternalMatchFinderResult nits
* test sum(matchlen) + sum(litlen) == srcSize in debug builds
* refExternalMatchFinder -> registerExternalMatchFinder
* C90 nit
* zstreamtest nits
* contrib nits
* contrib nits
* allow block splitter + external matchfinder, refactor
* add windowSize param
* add contrib/externalMatchfinder/README.md
* docs
* go back to old RLE heuristic because of the first block issue
* fix initializer element is not a constant expression
* ref contrib from zstd.h
* extremely pedantic compiler warning fix, meson fix, typo fix
* Additional docs on API limitations
* minor nits
* Refactor maxNbSeq calculation into a helper function
* Fix copyright
2022-12-28 16:45:14 -05:00
Yann Collet
6a9c525903
spec update : require minimum nb of literals for 4-streams mode
...
Reported by @shulib :
the specification for 4-streams mode
doesn't work when the amount of literals to compress is 5 bytes.
Extending it, it also doesn't work for sizes 1 or 2.
This patch updates the specification and the implementation
to require a minimum of 6 literals to trigger or accept the 4-streams mode.
The impact is expected to be a no-op :
the 4-streams mode is never triggered for such small quantity of literals anyway,
since it would be wasteful (it costs ~7.3 bytes more than single-stream mode).
An informal lower limit is set at ~256 bytes,
so the technical minimum is very far from this limit.
This is just meant for completeness of the specification.
2022-12-22 16:14:34 -08:00
W. Felix Handte
5d693cc38c
Coalesce Almost All Copyright Notices to Standard Phrasing
...
```
for f in $(find . \( -path ./.git -o -path ./tests/fuzz/corpora -o -path ./tests/regression/data-cache -o -path ./tests/regression/cache \) -prune -o -type f); do sed -i '/Copyright .* \(Yann Collet\)\|\(Meta Platforms\)/ s/Copyright .*/Copyright (c) Meta Platforms, Inc. and affiliates./' $f; done
git checkout HEAD -- build/VS2010/libzstd-dll/libzstd-dll.rc build/VS2010/zstd/zstd.rc tests/test-license.py contrib/linux-kernel/test/include/linux/xxhash.h examples/streaming_compression_thread_pool.c lib/legacy/zstd_v0*.c lib/legacy/zstd_v0*.h
nano ./programs/windres/zstd.rc
nano ./build/VS2010/zstd/zstd.rc
nano ./build/VS2010/libzstd-dll/libzstd-dll.rc
```
2022-12-20 12:52:34 -05:00
W. Felix Handte
8927f985ff
Update Copyright Headers 'Facebook' -> 'Meta Platforms'
...
```
for f in $(find . \( -path ./.git -o -path ./tests/fuzz/corpora \) -prune -o -type f);
do
sed -i 's/Facebook, Inc\./Meta Platforms, Inc. and affiliates./' $f;
done
```
2022-12-20 12:37:57 -05:00
Nick Terrell
358a237484
[api][visibility] Make the visibility macros more consistent
...
1. Follow the scheme introduced in PR #2501 for both `zdict.h` and `zstd_errors.h`.
2. If the `*_VISIBLE` macro isn't set, but the `*_VISIBILITY` macro is, use that.
Also make this change for `zstd.h`, since we probably shouldn't have changed
that macro name without backward compatibility in the first place.
3. Change all references to `*_VISIBILITY` to `*_VISIBLE`.
Fixes #3359 .
2022-12-16 12:54:45 -08:00
Yann Collet
b99ece96b9
converted checks into user validation generating error codes
...
had to create a new error code for this condition,
none of the existing ones were fitting enough.
2022-01-26 10:43:50 -08:00
Nick Terrell
09149beaf8
[1.5.0] Move zstd_errors.h and zdict.h to lib/ root
...
`zstd_errors.h` and `zdict.h` are public headers, so they deserve to be
in the root `lib/` directory with `zstd.h`, not mixed in with our private
headers.
2021-04-30 15:13:54 -07:00
