1
0
mirror of https://github.com/facebook/zstd.git synced 2025-07-06 15:45:37 +02:00
Commit Graph

1102 Commits

Author SHA1 Message Date
7083f79008 [bug] Fix dictContentType when reprocessing cdict
Conditions to trigger:
* CDict is loaded as raw content.
* CDict starts with the zstd dictionary magic number.
* The CDict is reprocessed (not attached or copied).
* The new API is used (streaming or `ZSTD_compress2()`).

Bug: The dictionary is loaded as a zstd dictionary, not a raw content
dictionary, because the dict content type is set to `ZSTD_dct_auto`.

Fix: Pass in the dictionary content type from cdict creation to the call
to `ZSTD_compress_insertDictionary()`.

Test: Added a test case that exposes the bug, and fixed the raw
content tests to not modify the `dictBuffer`, which makes all future
tests with the `dictBuffer` raw content, which doesn't seem intentional.
2020-10-12 12:46:10 -07:00
12541931fa Merge pull request #2328 from marxin/zstd-pool-api
Allow external creation of POOLs that can be shared.
2020-10-09 01:00:50 -07:00
6fdb0cb8d9 Merge pull request #2303 from senhuang42/let_cdict_take_clevel_priority
For ZSTD_compressStream2(), let cdict take compression level priority
2020-10-09 00:48:30 -07:00
b9c8033cde Define kNullRawSeqStore for every file 2020-10-07 19:02:41 -04:00
a6165c1b28 Change matchState_t::ldmSeqStore to pointer 2020-10-07 14:13:57 -04:00
10647924f1 Make function descriptions more accurate 2020-10-07 13:56:25 -04:00
37617e23d7 Correct matchLength calculation and remove unnecessary functions 2020-10-07 13:56:25 -04:00
ef823e0299 Remove rawSeqStore.base and add rawSeqStore.posInSequence 2020-10-07 13:56:25 -04:00
ea92fb3a68 Cleanups, add comments and explanations 2020-10-07 13:56:25 -04:00
6ccd97fc96 Fixed end of match boundary update issues 2020-10-07 13:56:25 -04:00
c8b8572b38 Adjustments to no longer segfault on nci 2020-10-07 13:56:25 -04:00
b684900a4a Allow external creation of POOLs that can be shared. 2020-10-07 12:44:33 +02:00
6a1e526ea7 [lib] Add ZSTD_COMPRESS_HEAPMODE tuning parameter 2020-09-24 19:42:04 -07:00
b841387218 [freestanding] Improve macro resolution to handle #if X 2020-09-24 19:42:04 -07:00
88fac5d514 Remove call to memset
The previous commit fixes the test so it errors on calls to mem*()
functions from <string.h>.
2020-09-24 19:42:04 -07:00
9261476b7d [lib] Wrap customMem xor checks in parens for readability
This clarifies operator precedence, and quiets cppcheck in
the Kernel Test Robot. I think this is a slight bonus to
readability, so I am accepting the suggestion.
2020-09-23 23:26:07 -07:00
200c960f1d Merge pull request #2311 from felixhandte/ddss-fix-cparam-derivation
Fix Compression Parameter Derivation Bugs Introduced by DDSS Changes
2020-09-18 14:02:14 -04:00
8930c6e551 Use ZSTD_CCtxParams_init() to Init CCtxParams, not memset()
Even if the discrepancies are at the moment benign, it's probably better to
standardize on using the one true initializer, rather than trying (and failing)
to correctly duplicate its behavior.
2020-09-17 12:15:33 -04:00
e8a44326fa Avoid Redundancy in ZSTD_initCDict_internal() Args; Don't Take CParams + CCtxParams 2020-09-17 12:08:36 -04:00
eee51a664a Fall Back if Derived CParams are Incompatible with DDSS; Refactor CDict Creation
Rewrite ZSTD_createCDict_advanced() as a wrapper around
ZSTD_createCDict_advanced2(). Evaluate whether to use DDSS mode *after* fully
resolving cparams. If not, fall back.
2020-09-15 18:01:08 -04:00
bc6521a6f6 Make ZSTD_createCDict_advanced2() cctxParams Arg Const 2020-09-15 14:06:10 -04:00
26a96a5b35 Do More Complete CParams Deduction in Non-DDSS Path of ZSTD_createCDict_advanced2
Call ZSTD_getCParamsFromCCtxParams() instead of ZSTD_getCParams_internal().
2020-09-15 13:57:43 -04:00
a2af804129 Pull CParam Override Logic into Helper 2020-09-15 13:38:05 -04:00
c91a0855f8 check endDirective in ZSTD_compressStream2()
fix #2297
also :
- `assert()` `endDirective` in `ZSTD_compressStream_internal()`, for debug mode
- add relevant tests
2020-09-14 10:56:08 -07:00
17b56f934e Coding style cleanup 2020-09-11 11:42:12 -04:00
801513b5e7 Modify params rather than cctx->requestedParams 2020-09-11 11:41:10 -04:00
0faefbf1b3 Make DDSS Selection Override ForceCopy Directive 2020-09-10 22:10:02 -04:00
13c5ec3e41 Only Allow Dedicated Dict Search for Dicts Loaded in 1 Chunk
The load algorithm requires we do it all in one go.
2020-09-10 22:10:02 -04:00
07793547e6 Fix Bug: Only Use DDSS Insertion on CDict MatchStates
Previously, if DDSS was enabled on a CCtx and a dictionary was inserted into
the CCtx, the CCtx MatchState would be filled as a DDSS struct, causing
segfaults etc. This changes the check to use whether the MatchState is marked
as using the DDSS (which is only ever set for CDict MatchStates), rather than
looking at the CCtxParams.
2020-09-10 18:51:52 -04:00
7b5d2f72ea Adjust Working Context Table Sizes Back Down 2020-09-10 18:51:52 -04:00
a3659fe1ef Make ZSTD_dedicatedDictSearch_getCParams Wrap ZSTD_getCParams
Fixes up bounds-checking, and lets us clean up what is at the moment an
unnecessary duplication of the default cparams tables.
2020-09-10 18:51:52 -04:00
5390fee4f7 Rename and Move DD_BLOG Constant to ZSTD_LAZY_DDSS_BUCKET_LOG 2020-09-10 18:51:52 -04:00
914bfe7ee4 Init CCtx's Local Dict with CCtxParams 2020-09-10 18:51:52 -04:00
db2aa25252 Decision for Whether to Attach Should be Based on CDict Config, not CCtx 2020-09-10 18:51:52 -04:00
f1b428fdac Rename enableDedicatedDictSearch to dedicatedDictSearch in MatchState
This makes it clear that not only is the feature allowed here, we're actually
using it, as opposed to the CCtxParam field, in which it's enabled, but we may
or may not be using it.
2020-09-10 18:51:52 -04:00
41012193ad Always Init CDict's enableDedicatedDictSearch Field 2020-09-10 18:51:52 -04:00
34b545acb0 Add a ZSTD_dedicatedDictSearch ZSTD_dictMode_e to Allow Const Propagation
Speed +1.5%.
2020-09-10 18:51:52 -04:00
beefdb0d3d Fix ZSTD_c_forceAttachDict Bounds 2020-09-10 18:51:52 -04:00
def62e2d3e Fix Compilation Warnings 2020-09-10 18:51:52 -04:00
9c628238d3 creating ZSTD_createCDict_advanced_internal 2020-09-10 18:51:52 -04:00
0a9787c3e1 changing to int for consistency 2020-09-10 18:51:52 -04:00
5d5507788d change method name for consistency 2020-09-10 18:51:52 -04:00
b30f71becf pass correct cparams 2020-09-10 18:51:52 -04:00
71fda0362f making cctxParams a pointer 2020-09-10 18:51:52 -04:00
628559d0e4 loading dict using new algorithm 2020-09-10 18:51:52 -04:00
31e581bf65 adding enableDedicatedDictSearch to matchState_t 2020-09-10 18:51:52 -04:00
75b6360036 adding ZSTD_createCDict_advanced2 to zstd.h 2020-09-10 18:51:52 -04:00
b7dddbe89b always attach dict when using dedicatedDictSearch 2020-09-10 18:51:52 -04:00
e36a373df4 adding dedicatedDictSearch cParams helper methods 2020-09-10 18:51:52 -04:00
f10d4e313c adding ZSTD_dedicatedDictSearch_defaultCParameters variable 2020-09-10 18:51:52 -04:00