krak/zstd
krak/zstd
1
0
Fork 0
mirror of https://github.com/facebook/zstd.git synced 2025-03-06 16:56:49 +02:00
Code Issues Releases 1 Activity
9,879 Commits 25 Branches 76 Tags
Commit Graph

629 Commits

Author SHA1 Message Date
Yann Collet
481a2e1010
Merge pull request #3403 from facebook/setCParams 
ZSTD_CCtx_setCParams
 2022-12-28 14:07:13 -08:00
Elliot Gorokhovsky
2a402626dd
External matchfinder API (#3333) 
* First building commit with sample matchfinder

* Set up ZSTD_externalMatchCtx struct

* move seqBuffer to ZSTD_Sequence*

* support non-contiguous dictionary

* clean up parens

* add clearExternalMatchfinder, handle allocation errors

* Add useExternalMatchfinder cParam

* validate useExternalMatchfinder cParam

* Disable LDM + external matchfinder

* Check for static CCtx

* Validate mState and mStateDestructor

* Improve LDM check to cover both branches

* Error API with optional fallback

* handle RLE properly for external matchfinder

* nit

* Move to a CDict-like model for resource ownership

* Add hidden useExternalMatchfinder bool to CCtx_params_s

* Eliminate malloc, move to cwksp allocation

* Handle CCtx reset properly

* Ensure seqStore has enough space for external sequences

* fix capitalization

* Add DEBUGLOG statements

* Add compressionLevel param to matchfinder API

* fix c99 issues and add a param combination error code

* nits

* Test external matchfinder API

* C90 compat for simpleExternalMatchFinder

* Fix some @nocommits and an ASAN bug

* nit

* nit

* nits

* forward declare copySequencesToSeqStore functions in zstd_compress_internal.h

* nit

* nit

* nits

* Update copyright headers

* Fix CMake zstreamtest build

* Fix copyright headers (again)

* typo

* Add externalMatchfinder demo program to make contrib

* Reduce memory consumption for small blockSize

* ZSTD_postProcessExternalMatchFinderResult nits

* test sum(matchlen) + sum(litlen) == srcSize in debug builds

* refExternalMatchFinder -> registerExternalMatchFinder

* C90 nit

* zstreamtest nits

* contrib nits

* contrib nits

* allow block splitter + external matchfinder, refactor

* add windowSize param

* add contrib/externalMatchfinder/README.md

* docs

* go back to old RLE heuristic because of the first block issue

* fix initializer element is not a constant expression

* ref contrib from zstd.h

* extremely pedantic compiler warning fix, meson fix, typo fix

* Additional docs on API limitations

* minor nits

* Refactor maxNbSeq calculation into a helper function

* Fix copyright
 2022-12-28 16:45:14 -05:00
Yann Collet
89342d1e07 New xp library symbol : ZSTD_CCtx_setCParams() 
Inspired by #3395,
offer a new capability to set all parameters defined in a ZSTD_compressionParameters structure
with a single symbol invocation
to improve user code brevity.
 2022-12-27 23:49:22 -08:00
Nick Terrell
40a7188130 Fix make clangbuild & add CI 
Fix the errors for:
* `-Wdocumentation`
* `-Wconversion` except `-Wsign-conversion`
 2022-12-21 17:31:04 -08:00
Nick Terrell
e4018c4e7f [docs] Clarify dictionary loading documentation 
Reinforce that loading a new dictionary clears the current dictionary.
Except for the multiple-ddict mode.
 2022-12-20 11:10:49 -08:00
W. Felix Handte
5d693cc38c Coalesce Almost All Copyright Notices to Standard Phrasing 
```
for f in $(find . \( -path ./.git -o -path ./tests/fuzz/corpora -o -path ./tests/regression/data-cache -o -path ./tests/regression/cache \) -prune -o -type f); do sed -i '/Copyright .* \(Yann Collet\)\|\(Meta Platforms\)/ s/Copyright .*/Copyright (c) Meta Platforms, Inc. and affiliates./' $f; done

git checkout HEAD -- build/VS2010/libzstd-dll/libzstd-dll.rc build/VS2010/zstd/zstd.rc tests/test-license.py contrib/linux-kernel/test/include/linux/xxhash.h examples/streaming_compression_thread_pool.c lib/legacy/zstd_v0*.c lib/legacy/zstd_v0*.h
nano ./programs/windres/zstd.rc
nano ./build/VS2010/zstd/zstd.rc
nano ./build/VS2010/libzstd-dll/libzstd-dll.rc
```
 2022-12-20 12:52:34 -05:00
W. Felix Handte
8927f985ff Update Copyright Headers 'Facebook' -> 'Meta Platforms' 
```
for f in $(find . \( -path ./.git -o -path ./tests/fuzz/corpora \) -prune -o -type f);
do
  sed -i 's/Facebook, Inc\./Meta Platforms, Inc. and affiliates./' $f;
done
```
 2022-12-20 12:37:57 -05:00
Yann Collet
2f4238e47a make ZSTD_DECOMPRESSBOUND() compatible with input size 0 
for environments with stringent compilation warnings.
 2022-12-16 16:05:39 -08:00
Yann Collet
51355e1f70
Merge pull request #3362 from facebook/compressBound 
check potential overflow of compressBound()
 2022-12-16 14:22:22 -08:00
Nick Terrell
358a237484 [api][visibility] Make the visibility macros more consistent 
1. Follow the scheme introduced in PR #2501 for both `zdict.h` and `zstd_errors.h`.
2. If the `*_VISIBLE` macro isn't set, but the `*_VISIBILITY` macro is, use that.
   Also make this change for `zstd.h`, since we probably shouldn't have changed
   that macro name without backward compatibility in the first place.
3. Change all references to `*_VISIBILITY` to `*_VISIBLE`.

Fixes #3359.
 2022-12-16 12:54:45 -08:00
Yann Collet
97f63ce2b5 added unit tests for compressBound() 
and rephrased the code documentation, as suggested by @terrelln
 2022-12-16 12:35:14 -08:00
Yann Collet
45ed0df18a check potential overflow of compressBound() 
fixed #3323, reported by @nigeltao

Completed documentation around this risk
(which is largely theoretical,
I can't see that happening in any "real world" scenario,
but an erroneous @srcSize value could indeed trigger it).
 2022-12-15 15:23:15 -08:00
Elliot Gorokhovsky
e1e82f74f1 Reserve two fields in ZSTD_frameHeader 2022-12-13 17:45:05 -08:00
Fangrui Song
5635827ede Move ZSTD_DEPRECATED before ZSTDLIB_API/ZSTDLIB_STATIC_API 
Clang doesn't allow [[deprecated(...)]] attribute after __attribute__.
Move [[deprecated(...)]] before __attribute__ to fix C++14/C++17 uses
with Clang.

Fix #3250
 2022-09-22 12:30:44 -07:00
Yann Collet
434ffe979c minor: refactor publication of ZSTD_copyCCtx() 
for improved clarity
 2022-09-22 11:14:21 -07:00
Danielle Rozenblit
aa82998821 add sequence bound function 2022-09-09 12:34:25 -07:00
Dimitri Papadopoulos
0015308c0f
Fix typos found by codespell 2022-09-08 23:17:00 +02:00
Elliot Gorokhovsky
ef60302af9
Merge pull request #3230 from grossws/fix3229-docs 
Add description for ZSTD_decompressStream and ZSTD_initDStream
 2022-08-16 12:48:23 -04:00
Konstantin Gribov
1c847e2e32 Add description for ZSTD_decompressStream and ZSTD_initDStream 
With that these functions become visible in generated docs.

Fixes #3229
 2022-08-08 18:02:50 +03:00
Felix Handte
7e6278a706
Merge pull request #3196 from mileshu/dev 
[T124890272] Mark 2 Obsolete Functions(ZSTD_copy*Ctx) Deprecated in Zstd
 2022-08-02 12:34:04 -04:00
Miles HU
c450f9f952 [T124890272] Mark 2 Obsolete Functions(ZSTD_copy*Ctx) Deprecated in Zstd 
The discussion for this task is here: facebook/zstd#3128.

This task can probably be scoped to the first part: marking these functions deprecated.
We'll later look at removal when we roll out v1.6.0.
 2022-08-01 22:45:52 -07:00
Nick Terrell
0f4fd28a64
Deprecate ZSTD_getDecompressedSize() (#3225) 
Fixes #3158.

Mark ZSTD_getDecompressedSize() as deprecated and replaced by ZSTD_getFrameContentSize().
 2022-08-01 11:52:14 -07:00
Miles HU
a5655e4017 Revert "T119975957" 
This reverts commit 962746edffa5340315136af34ac3331eba82c3c8.
 2022-07-12 11:17:25 -07:00
Miles HU
962746edff T119975957 
Signed-off-by: Miles HU <yuanpu@fb.com>
 2022-07-08 15:01:36 -07:00
Elliot Gorokhovsky
5c382bf110 1.5.3 version bump 2022-06-29 14:45:53 -04:00
Elliot Gorokhovsky
93b89fb24b Add docs 2022-06-22 16:13:07 -04:00
Elliot Gorokhovsky
2a128110d0 Add prefetchCDictTables CCtxParam 2022-06-22 16:13:07 -04:00
Dominique Pelle
b772f53952 Typo and grammar fixes 2022-03-12 08:58:04 +01:00
Yann Collet
bad7f82300
Merge pull request #2974 from facebook/fix2966_part3 
Lazy parameters adaptation (part 1 - ZSTD_c_stableInBuffer)
 2022-01-27 06:14:04 -08:00
Yann Collet
7616e39f3b adding traces to better track processing of literals 2022-01-26 14:47:21 -08:00
Yann Collet
27d336b099 minor behavior refinements 
specifically, there is no obligation to start streaming compression with pos=0.
stableSrc mode is now compatible with this setup.
 2022-01-26 10:31:24 -08:00
Yann Collet
37b87add7a make stableSrc compatible with regular streaming API 
including flushStream().

Now the only condition is for `input.size` to continuously grow.
 2022-01-26 10:31:24 -08:00
Yann Collet
c0c5ffa973 streaming compression : lazy parameter adaptation with stable input 
effectively makes ZSTD_c_stableInput compatible ZSTD_compressStream()
and zstd_e_continue operation mode.
 2022-01-26 10:31:24 -08:00
Yann Collet
87dcd3326a fix sequence compression API in Explicit Delimiter mode 2022-01-25 13:33:41 -08:00
Tom Rix
2b957afec7 cleanup double word in comment. 
Remove the second 'a' and 'into'

Signed-off-by: Tom Rix <trix@redhat.com>
 2022-01-24 12:43:39 -08:00
W. Felix Handte
46ad9377e8 Bump Version Number to 1.5.2 2022-01-07 14:14:26 -05:00
Yann Collet
3f64b31585 Merge branch 'dev' into tomerge2051 2021-12-01 15:29:49 -08:00
Dimitris Apostolou
ebbd675998
Fix typos 2021-11-13 10:04:04 +02:00
Martin Liska
1c2b02eee9 Support thread pool section in HTML documentation. 2021-10-15 18:35:39 +02:00
Abshar Mohammed Aslam
54a888b57b
Fix typo 2021-09-23 21:54:38 +04:00
sen
1e99d36361
Merge pull request #2788 from senhuang42/param_switch 
Use new paramSwitch enum for row matchfinder and block splitter
 2021-09-22 13:27:55 -04:00
senhuang42
b5c35d7ea3 Use new paramSwitch enum for LCM, row matchfinder, and block splitter 2021-09-21 14:22:02 -04:00
Nick Terrell
a5f2c45528 Huffman ASM 2021-09-20 14:46:43 -07:00
Yann Collet
2de42174bb make ZSTD_HASHLOG3_MAX private 
This is an implementation detail,
it doesn't belong to public space (zstd.h).
 2021-08-20 09:52:42 -07:00
Binh Vo
9d9f7680f8 Add API for fetching skippable frame content 2021-06-14 16:01:28 -04:00
Stephen Kitt
e81d567547
Distinguish static symbols, allow hiding them 
Even with -fvisibility=hidden added to CFLAGS, any symbol which is
given a default visibility attribute ends up exported in the dynamic
library. This happens through zstd_internal.h which defines
..._STATIC_LINKING_ONLY before including various header files, and is
included for example in lib/common/pool.c.

To avoid this, this patch distinguishes static and non-static APIs, by
using ZSTDLIB_API only for the latter, and introducing
ZSTDLIB_STATIC_API for the former. For now, both are exported, but
non-static APIs can be hidden by overriding the definition
ZSTDLIB_STATIC_API. lib/Makefile is modified to allow this using

	make CPPFLAGS_DYNLIB=-DZSTDLIB_STATIC_API=ZSTDLIB_HIDDEN

In addition, API declarations are dropped from zstd_compress.c (they
aren't needed there).

Signed-off-by: Stephen Kitt <steve@sk2.org>
 2021-05-14 19:41:59 +02:00
sen
c730b8c5a3
Remove const data members in threadpooltest payload (#2639) (#2640) 2021-05-12 16:09:48 -04:00
sen
9c23ea9e2b
Bump version to 1.5.0, rebuild documentation (#2634) 2021-05-11 16:32:09 -04:00
sen
698f261b35
[1.5.0] Deprecate some functions (#2582) 
* Add deprecated macro to zstd.h, mark certain functions as deprecated

* Remove ZSTD_compress.c dependencies on deprecated functions
 2021-05-06 17:59:32 -04:00
Nick Terrell
172b4b6ac4 [lib] Add ZSTD_c_deterministicRefPrefix 
This flag forces zstd to always load the prefix in ext-dict mode, even
if it happens to be contiguous, to force determinism. It also applies to
dictionaries that are re-processed.

A determinism test case is also added, which fails without
`ZSTD_c_deterministicRefPrefix` and passes with it set.

Question: Should this be the default behavior? It isn't in this PR.
 2021-05-05 18:49:56 -07:00