1
0
mirror of https://github.com/facebook/zstd.git synced 2025-07-04 14:48:45 +02:00
Commit Graph

880 Commits

Author SHA1 Message Date
a54c86cfc6 defined a minimum negative level
which can be probed using new function ZSTD_minCLevel().

Also : redefined ZSTD_TARGETLENGTH_MIN/MAX for consistency

used the opportunity to bump version number to v1.3.6
2018-09-20 16:52:03 -07:00
2f78228f65 Merge branch 'dev' into adapt 2018-09-19 12:43:42 -07:00
31ebb26945 Merge pull request #1301 from terrelln/lit-size
[zstd] Fix seqStore growth
2018-08-28 17:10:25 -07:00
5e580de6da [zstd] Fix seqStore growth
We could undersize the literals buffer by up to 11 bytes,
due to a combination of 2 bugs:
* The literals buffer didn't have `WILDCOPY_OVERLENGTH` extra
  space, like it is supposed to.
* We didn't check the literals buffer size in `ZSTD_sufficientBuff()`.
2018-08-28 13:24:44 -07:00
b37a0a6bde Merge pull request #1298 from facebook/bench
Refactored bench.c
2018-08-28 12:25:02 -07:00
af23d39eb8 Merge pull request #1297 from felixhandte/check-offset-table
Fix Missing Offset Table Check
2018-08-24 17:36:44 -07:00
37f17ee237 Mark Repeated Offset Table as Needing Check 2018-08-24 14:33:34 -07:00
e34e917655 Fix compiler warning 2018-08-23 17:48:06 -07:00
924944e471 [zstd] Reuse the ZSTD_CCtx more often with small data. 2018-08-23 17:48:06 -07:00
2e45badff4 refactored bench.c
for clarity and safety, especially at interface level
2018-08-23 14:21:18 -07:00
c71c4f23d7 fix "unused parameter" in single-thread mode
within newly added ZSD_toFlushNow()
2018-08-20 11:40:10 -07:00
105677c6db created ZSTDMT_toFlushNow()
tells in a non-blocking way if there is something ready to flush right now.
only works with multi-threading for the time being.

Useful to know if flush speed will be limited by lack of production.
2018-08-17 18:11:54 -07:00
6e66bbf5dd fixed several minor issues detected by scan-build
only notable one :
writeNCount() resists better vs invalid distributions
(though it should never happen within zstd anyway)
2018-08-14 16:55:35 -07:00
3e4617ef54 frameProgression reports nbActiveWorkers and output flushed 2018-08-14 11:49:25 -07:00
e7a49c6683 introduced command --adapt 2018-08-11 20:48:06 -07:00
2dd76037be zstd cli can increase level when input is too slow 2018-08-09 15:51:30 -07:00
2ca7c69167 Fix CDict Attachment to Handle CDicts with Non-Zero Starts
CDicts were previously guaranteed to be generated with `lowLimit=dictLimit=0`.
This is no longer true, and so the old length and index calculations are no
longer valid. This diff fixes them to handle non-zero start indices in CDicts.
2018-08-07 18:14:14 -07:00
aade1e5904 Merge branch 'dev' into fix1241 2018-07-30 16:30:35 +02:00
9597b438e9 fix #1241
Ensure that first input position is valid for a match
even during first usage of context
by starting reference at 1
(avoiding the problematic 0).
2018-07-17 18:52:57 +02:00
6d222c437c Set requestedParams in ZSTD_initCStream*()
The correct parameters are used once, but once `ZSTD_resetCStream()` is
called the default parameters (level 3) are used. Fix this by setting
`requestedParams` in the `ZSTD_initCStream*()` functions.

The added tests both fail before this patch and pass after.
2018-07-12 18:35:55 -07:00
4489daec09 slightly adjusted default-distribution threshold
depending on strategy.
fast favors faster compression and decompression speeds.
2018-06-26 20:10:45 -07:00
3b53bfe4f3 Merge pull request #1200 from felixhandte/zstd-attach-dict-pref
Add CCtx Param Controlling Dict Attachment Behavior
2018-06-25 12:42:31 -07:00
3934e010a2 Merge pull request #1197 from facebook/poolResize
Thread Pool resize
2018-06-22 14:20:07 -07:00
1d5648ca10 Merge pull request #1196 from felixhandte/zstd-btopt-in-place-dict
ZSTD_btopt: Support Searching the Dictionary Context In-Place
2018-06-22 11:53:23 -07:00
01bb1c1016 Add CCtx Param Controlling Dict Attachment Behavior 2018-06-21 17:29:25 -04:00
066fbbfe1c make zstdmt resize its context
when nbThreads change.

Technically, it only expands.
But when instructed to use less threads,
the thread pool will limit nb of concurrent threads.
2018-06-19 17:28:56 -07:00
6768cf53fd Merge pull request #1190 from terrelln/ldm-adjust
Adjust advanced parameters to source size
2018-06-19 14:40:56 -07:00
de639502aa Update Dict Attachment Cut-Offs 2018-06-19 15:36:13 -04:00
f0a13bcd68 Make Sure Position 0 Gets Into the Tree 2018-06-19 15:10:06 -04:00
ccbf067973 Add _dictMatchState Functions 2018-06-19 13:01:21 -04:00
93c3184d44 Attach Dicts when Using ZSTD_btopt and ZSTD_btultra 2018-06-19 13:01:21 -04:00
3841dbac84 Adjust advanced parameters to source size
In the new advanced API, adjust the parameters even if they are explicitly
set. This mainly applies to the `windowLog`, and accordingly the `hashLog`
and `chainLog`, when the source size is known.
2018-06-18 15:49:31 -07:00
e30f13bde0 Merge pull request #1185 from felixhandte/zstd-btlazy-in-place-dict
ZSTD_btlazy2: Support Searching the Dictionary Context In-Place
2018-06-18 13:29:44 -07:00
9698d2fb72 Merge pull request #1189 from facebook/hist
histogram module
2018-06-14 20:39:52 -04:00
2d76defbfe grouped all histogram functions into hist.c
renamed functions with HIST_* prefix
2018-06-13 19:49:31 -04:00
ace9cfa950 Attach Dicts when Using ZSTD_btlazy2 2018-06-13 16:06:28 -04:00
d53a04211c Update Dictionary Attachment Cutoff Values Again 2018-06-13 14:58:36 -04:00
c87dd2121d Update Dictionary Attachment Cutoff Values 2018-06-13 14:58:36 -04:00
211a61b69b Focus on Non-BT Impls for the Moment 2018-06-13 14:58:36 -04:00
33e2240fac Attach Dict When Using ZSTD_lazy Strategies 2018-06-13 14:58:36 -04:00
90cfc799e5 Add _dictMatchState Stubs for ZSTD_lazy Functions 2018-06-13 14:58:36 -04:00
b2632bcf6c Merge pull request #1174 from duc0/document_default_level
Expose ZSTD_CLEVEL_DEFAULT and update documentation
2018-06-12 12:09:01 -07:00
e34c000e44 Expose ZSTD_CLEVEL_DEFAULT and update documentation 2018-06-08 11:33:44 -07:00
3050733042 Merge branch 'dev' into negLevels 2018-06-07 15:51:35 -07:00
c2c47e24e0 support targetlen==0 with strategy==ZSTD_fast
to mean "normal compression",
targetlen >= 1 now means "disable huffman compression of literals"
2018-06-07 15:49:01 -07:00
a57b4df85f removed literalCompression directive
in this version, literal compression is always disabled for ZSTD_fast strategy.

Performance parity between ZSTD_compress_advanced() and ZSTD_compress_generic()
2018-06-07 15:24:12 -07:00
8537bfd85c fuzzer: make negative compression level fail
result of ZSTD_compress_advanced()
is different from ZSTD_compress_generic()
when using negative compression levels
because the disabling of huffman compression is not passed in parameters.
2018-06-07 15:12:13 -07:00
e3c42c739b clean ZSTD_compress() initialization
The (pretty old) code inside ZSTD_compress()
was making some pretty bold assumptions
on what's inside a CCtx and how to init it.

This is pretty fragile by design.
CCtx content evolve.
Knowledge of how to handle that should be concentrate in one place.

A side effect of this strategy
is that ZSTD_compress() wouldn't check for BMI2 capability,
and is therefore missing out some potential speed opportunity.

This patch makes ZSTD_compress() use
the same initialization and release functions
as the normal creator / destructor ones.

Measured on my laptop, with a custom version of bench
manually modified to use ZSTD_compress() (instead of the advanced API) :
This patch :
 1#silesia.tar       : 211984896 ->  73651053 (2.878), 312.2 MB/s , 723.8 MB/s
 2#silesia.tar       : 211984896 ->  70163650 (3.021), 226.2 MB/s , 649.8 MB/s
 3#silesia.tar       : 211984896 ->  66996749 (3.164), 169.4 MB/s , 636.7 MB/s
 4#silesia.tar       : 211984896 ->  65998319 (3.212), 136.7 MB/s , 619.2 MB/s
dev branch :
 1#silesia.tar       : 211984896 ->  73651053 (2.878), 291.7 MB/s , 727.5 MB/s
 2#silesia.tar       : 211984896 ->  70163650 (3.021), 216.2 MB/s , 655.7 MB/s
 3#silesia.tar       : 211984896 ->  66996749 (3.164), 162.2 MB/s , 633.1 MB/s
 4#silesia.tar       : 211984896 ->  65998319 (3.212), 130.6 MB/s , 618.6 MB/s
2018-06-07 14:05:25 -07:00
f1ea383f45 context can be sized down even with constant parameters
when parameters are "equivalent",
the context is re-used in continue mode,
hence needed workspace size is not recalculated.
This incidentally also evades the size-down check and action.

This patch intercepts the "continue mode"
so that the size-down check and action is actually triggered.
2018-06-06 15:04:12 -07:00
e5e17d009f changed member name to workSpaceOversizedDuration 2018-06-06 15:00:27 -07:00