1
0
mirror of https://github.com/facebook/zstd.git synced 2025-07-04 22:58:46 +02:00
Commit Graph

698 Commits

Author SHA1 Message Date
1f188ae655 Move Asserts into Function to Avoid Unused Function Warning 2018-09-28 17:12:54 -07:00
7212b5e5c2 Move Match State CParams Setting into resetCCtx and continueCCtx 2018-09-28 17:12:54 -07:00
01e34d365b Strengthen Assertion to Assert Equality 2018-09-28 17:12:53 -07:00
50cc1cf4d5 Remove CParams Arg from ZSTD_ldm_blockCompress 2018-09-28 17:12:53 -07:00
dcdf437fed Also Remove CParams from Table Filling Functions' Args 2018-09-28 17:10:42 -07:00
3483f89101 Also Assert Equivalency When Filling MatchState with Prefix 2018-09-28 17:10:42 -07:00
6cb2454646 Remove CParams from Block Compressor Functions' Args 2018-09-28 17:10:42 -07:00
03103269de Assert ctx and ms cparams Equivalency 2018-09-28 17:10:42 -07:00
4e3ecee9ed Remove cParams from CDict 2018-09-28 17:10:42 -07:00
76ef87ed9d Add ZSTD_compressionParameters to ZSTD_matchState_t 2018-09-28 17:10:42 -07:00
a180ea07c4 Restore ZSTD_noCompressBlock() for clarity 2018-09-27 16:06:02 -07:00
f2d6db45cd [zstd] Add -Wmissing-prototypes 2018-09-27 15:24:48 -07:00
2a5cd8535a Merge pull request #1342 from facebook/fixcatyd
fix : huge (>4GB) chain of blocks
2018-09-27 10:20:14 -07:00
404a7bfed0 moved again overflow correction
cannot work from within ZSTD_compressBlock()
2018-09-26 18:06:53 -07:00
0e2dbac18a changed overflow correction place
keep one in compress_frameChunk(),
so that it's tested at every loop
in case some user simply some large mulit-GB input in a single invocation.

Add one in ZSTD_compressBlock(),
since compressBlock() explicitly skips frameChunk().
2018-09-26 15:35:38 -07:00
f98c69d77c fix : huge (>4GB) stream of blocks
experimental function ZSTD_compressBlock() is designed for very small data in mind,
for situation where saving the ~12 bytes of frame header can actually make a difference.

Some systems though may have to deal with small and large data entangled.
If it's larger than a block (> 128KB), compressBlock() cannot compress them in one round.

That's why it's possible to compress in multiple rounds.
This is a chain of compressed blocks.

Some users push this capability to the limit, encoding gigantic chain of blocks.
On crossing the 4GB limit, some internal overflow occurs.

This fix moves the overflow correction mechanism higher in the call chain,
so that it's applied also to gigantic chains of blocks.

Added a test case in fuzzer.c, which crashes before the fix, and pass now.
2018-09-26 14:24:28 -07:00
04f47bbdd2 Merge branch 'dev' into adapt 2018-09-24 16:56:45 -07:00
32b7cf1bcf fixed tautological tests
involving ZSTD_TARGETLENGTH_MIN (== 0)
2018-09-21 15:04:43 -07:00
a54c86cfc6 defined a minimum negative level
which can be probed using new function ZSTD_minCLevel().

Also : redefined ZSTD_TARGETLENGTH_MIN/MAX for consistency

used the opportunity to bump version number to v1.3.6
2018-09-20 16:52:03 -07:00
2f78228f65 Merge branch 'dev' into adapt 2018-09-19 12:43:42 -07:00
31ebb26945 Merge pull request #1301 from terrelln/lit-size
[zstd] Fix seqStore growth
2018-08-28 17:10:25 -07:00
5e580de6da [zstd] Fix seqStore growth
We could undersize the literals buffer by up to 11 bytes,
due to a combination of 2 bugs:
* The literals buffer didn't have `WILDCOPY_OVERLENGTH` extra
  space, like it is supposed to.
* We didn't check the literals buffer size in `ZSTD_sufficientBuff()`.
2018-08-28 13:24:44 -07:00
b37a0a6bde Merge pull request #1298 from facebook/bench
Refactored bench.c
2018-08-28 12:25:02 -07:00
af23d39eb8 Merge pull request #1297 from felixhandte/check-offset-table
Fix Missing Offset Table Check
2018-08-24 17:36:44 -07:00
37f17ee237 Mark Repeated Offset Table as Needing Check 2018-08-24 14:33:34 -07:00
e34e917655 Fix compiler warning 2018-08-23 17:48:06 -07:00
924944e471 [zstd] Reuse the ZSTD_CCtx more often with small data. 2018-08-23 17:48:06 -07:00
2e45badff4 refactored bench.c
for clarity and safety, especially at interface level
2018-08-23 14:21:18 -07:00
c71c4f23d7 fix "unused parameter" in single-thread mode
within newly added ZSD_toFlushNow()
2018-08-20 11:40:10 -07:00
105677c6db created ZSTDMT_toFlushNow()
tells in a non-blocking way if there is something ready to flush right now.
only works with multi-threading for the time being.

Useful to know if flush speed will be limited by lack of production.
2018-08-17 18:11:54 -07:00
6e66bbf5dd fixed several minor issues detected by scan-build
only notable one :
writeNCount() resists better vs invalid distributions
(though it should never happen within zstd anyway)
2018-08-14 16:55:35 -07:00
3e4617ef54 frameProgression reports nbActiveWorkers and output flushed 2018-08-14 11:49:25 -07:00
e7a49c6683 introduced command --adapt 2018-08-11 20:48:06 -07:00
2dd76037be zstd cli can increase level when input is too slow 2018-08-09 15:51:30 -07:00
2ca7c69167 Fix CDict Attachment to Handle CDicts with Non-Zero Starts
CDicts were previously guaranteed to be generated with `lowLimit=dictLimit=0`.
This is no longer true, and so the old length and index calculations are no
longer valid. This diff fixes them to handle non-zero start indices in CDicts.
2018-08-07 18:14:14 -07:00
aade1e5904 Merge branch 'dev' into fix1241 2018-07-30 16:30:35 +02:00
9597b438e9 fix #1241
Ensure that first input position is valid for a match
even during first usage of context
by starting reference at 1
(avoiding the problematic 0).
2018-07-17 18:52:57 +02:00
6d222c437c Set requestedParams in ZSTD_initCStream*()
The correct parameters are used once, but once `ZSTD_resetCStream()` is
called the default parameters (level 3) are used. Fix this by setting
`requestedParams` in the `ZSTD_initCStream*()` functions.

The added tests both fail before this patch and pass after.
2018-07-12 18:35:55 -07:00
4489daec09 slightly adjusted default-distribution threshold
depending on strategy.
fast favors faster compression and decompression speeds.
2018-06-26 20:10:45 -07:00
3b53bfe4f3 Merge pull request #1200 from felixhandte/zstd-attach-dict-pref
Add CCtx Param Controlling Dict Attachment Behavior
2018-06-25 12:42:31 -07:00
3934e010a2 Merge pull request #1197 from facebook/poolResize
Thread Pool resize
2018-06-22 14:20:07 -07:00
1d5648ca10 Merge pull request #1196 from felixhandte/zstd-btopt-in-place-dict
ZSTD_btopt: Support Searching the Dictionary Context In-Place
2018-06-22 11:53:23 -07:00
01bb1c1016 Add CCtx Param Controlling Dict Attachment Behavior 2018-06-21 17:29:25 -04:00
066fbbfe1c make zstdmt resize its context
when nbThreads change.

Technically, it only expands.
But when instructed to use less threads,
the thread pool will limit nb of concurrent threads.
2018-06-19 17:28:56 -07:00
6768cf53fd Merge pull request #1190 from terrelln/ldm-adjust
Adjust advanced parameters to source size
2018-06-19 14:40:56 -07:00
de639502aa Update Dict Attachment Cut-Offs 2018-06-19 15:36:13 -04:00
f0a13bcd68 Make Sure Position 0 Gets Into the Tree 2018-06-19 15:10:06 -04:00
ccbf067973 Add _dictMatchState Functions 2018-06-19 13:01:21 -04:00
93c3184d44 Attach Dicts when Using ZSTD_btopt and ZSTD_btultra 2018-06-19 13:01:21 -04:00
3841dbac84 Adjust advanced parameters to source size
In the new advanced API, adjust the parameters even if they are explicitly
set. This mainly applies to the `windowLog`, and accordingly the `hashLog`
and `chainLog`, when the source size is known.
2018-06-18 15:49:31 -07:00