1
0
mirror of https://github.com/facebook/zstd.git synced 2025-07-05 15:09:03 +02:00
Commit Graph

1275 Commits

Author SHA1 Message Date
7212b5e5c2 Move Match State CParams Setting into resetCCtx and continueCCtx 2018-09-28 17:12:54 -07:00
01e34d365b Strengthen Assertion to Assert Equality 2018-09-28 17:12:53 -07:00
50cc1cf4d5 Remove CParams Arg from ZSTD_ldm_blockCompress 2018-09-28 17:12:53 -07:00
14764de49f Stop Separately Passing CParams in ZSTD_lazy Internal Functions 2018-09-28 17:12:53 -07:00
97149f22c3 Stop Separately Passing CParams in ZSTD_opt Internal Functions 2018-09-28 17:10:42 -07:00
dcdf437fed Also Remove CParams from Table Filling Functions' Args 2018-09-28 17:10:42 -07:00
3483f89101 Also Assert Equivalency When Filling MatchState with Prefix 2018-09-28 17:10:42 -07:00
6cb2454646 Remove CParams from Block Compressor Functions' Args 2018-09-28 17:10:42 -07:00
03103269de Assert ctx and ms cparams Equivalency 2018-09-28 17:10:42 -07:00
4e3ecee9ed Remove cParams from CDict 2018-09-28 17:10:42 -07:00
76ef87ed9d Add ZSTD_compressionParameters to ZSTD_matchState_t 2018-09-28 17:10:42 -07:00
6391cd1030 [zstd] Fix newly added test case 2018-09-28 12:09:28 -07:00
a180ea07c4 Restore ZSTD_noCompressBlock() for clarity 2018-09-27 16:06:02 -07:00
f2d6db45cd [zstd] Add -Wmissing-prototypes 2018-09-27 15:24:48 -07:00
2a5cd8535a Merge pull request #1342 from facebook/fixcatyd
fix : huge (>4GB) chain of blocks
2018-09-27 10:20:14 -07:00
404a7bfed0 moved again overflow correction
cannot work from within ZSTD_compressBlock()
2018-09-26 18:06:53 -07:00
0e2dbac18a changed overflow correction place
keep one in compress_frameChunk(),
so that it's tested at every loop
in case some user simply some large mulit-GB input in a single invocation.

Add one in ZSTD_compressBlock(),
since compressBlock() explicitly skips frameChunk().
2018-09-26 15:35:38 -07:00
f98c69d77c fix : huge (>4GB) stream of blocks
experimental function ZSTD_compressBlock() is designed for very small data in mind,
for situation where saving the ~12 bytes of frame header can actually make a difference.

Some systems though may have to deal with small and large data entangled.
If it's larger than a block (> 128KB), compressBlock() cannot compress them in one round.

That's why it's possible to compress in multiple rounds.
This is a chain of compressed blocks.

Some users push this capability to the limit, encoding gigantic chain of blocks.
On crossing the 4GB limit, some internal overflow occurs.

This fix moves the overflow correction mechanism higher in the call chain,
so that it's applied also to gigantic chains of blocks.

Added a test case in fuzzer.c, which crashes before the fix, and pass now.
2018-09-26 14:24:28 -07:00
04f47bbdd2 Merge branch 'dev' into adapt 2018-09-24 16:56:45 -07:00
c484345a82 Merge branch 'mingw' into adapt 2018-09-21 16:00:46 -07:00
bfff4f4809 ensure all writes to job->cSize are mutex protected
even when reporting errors,
using a macro for code brevity, as suggested by @terrelln,
2018-09-21 16:00:39 -07:00
32b7cf1bcf fixed tautological tests
involving ZSTD_TARGETLENGTH_MIN (== 0)
2018-09-21 15:04:43 -07:00
c044345f8f Merge branch 'mingw' into minclevel 2018-09-21 14:56:57 -07:00
de6c75e4e5 Merge pull request #1318 from felixhandte/shadow-dict-matches
Don't Search Dictionary Context When Working Context Search Resulted in Mismatch
2018-09-21 12:15:33 -07:00
a54c86cfc6 defined a minimum negative level
which can be probed using new function ZSTD_minCLevel().

Also : redefined ZSTD_TARGETLENGTH_MIN/MAX for consistency

used the opportunity to bump version number to v1.3.6
2018-09-20 16:52:03 -07:00
7992942d66 fixed complex tsan issue
when job->consumed == job->src.size , compression job is presumed completed,
so it must be the very last action done in worker thread.
2018-09-20 13:47:31 -07:00
6b07a66aec fixed minor reporting discrepancy in MT mode 2018-09-19 16:30:55 -07:00
ca02ebee07 removed static variables
so that --adapt can work on multiple input files too
2018-09-19 15:25:50 -07:00
2f78228f65 Merge branch 'dev' into adapt 2018-09-19 12:43:42 -07:00
18b4a1da61 Fix clang build
Fix dixygen comment
Fix clang binary path
2018-09-16 10:27:02 +09:00
b76c888497 ZSTD_dfast: Don't Search Dict Context When Mismatch Was Found 2018-09-14 15:24:25 -07:00
b048af5999 ZSTD_fast: Don't Search Dict Context When Mismatch Was Found 2018-09-14 15:23:35 -07:00
31ebb26945 Merge pull request #1301 from terrelln/lit-size
[zstd] Fix seqStore growth
2018-08-28 17:10:25 -07:00
5e580de6da [zstd] Fix seqStore growth
We could undersize the literals buffer by up to 11 bytes,
due to a combination of 2 bugs:
* The literals buffer didn't have `WILDCOPY_OVERLENGTH` extra
  space, like it is supposed to.
* We didn't check the literals buffer size in `ZSTD_sufficientBuff()`.
2018-08-28 13:24:44 -07:00
b37a0a6bde Merge pull request #1298 from facebook/bench
Refactored bench.c
2018-08-28 12:25:02 -07:00
af23d39eb8 Merge pull request #1297 from felixhandte/check-offset-table
Fix Missing Offset Table Check
2018-08-24 17:36:44 -07:00
37f17ee237 Mark Repeated Offset Table as Needing Check 2018-08-24 14:33:34 -07:00
e34e917655 Fix compiler warning 2018-08-23 17:48:06 -07:00
924944e471 [zstd] Reuse the ZSTD_CCtx more often with small data. 2018-08-23 17:48:06 -07:00
2e45badff4 refactored bench.c
for clarity and safety, especially at interface level
2018-08-23 14:21:18 -07:00
c71c4f23d7 fix "unused parameter" in single-thread mode
within newly added ZSD_toFlushNow()
2018-08-20 11:40:10 -07:00
105677c6db created ZSTDMT_toFlushNow()
tells in a non-blocking way if there is something ready to flush right now.
only works with multi-threading for the time being.

Useful to know if flush speed will be limited by lack of production.
2018-08-17 18:11:54 -07:00
1515f0bb0d fixed more issues detected by recent version of scan-build
test run on Linux
2018-08-16 15:20:25 -07:00
3692c31598 Merge branch 'dev' into scanbuild 2018-08-15 13:50:49 -07:00
6e66bbf5dd fixed several minor issues detected by scan-build
only notable one :
writeNCount() resists better vs invalid distributions
(though it should never happen within zstd anyway)
2018-08-14 16:55:35 -07:00
3e4617ef54 frameProgression reports nbActiveWorkers and output flushed 2018-08-14 11:49:25 -07:00
e7a49c6683 introduced command --adapt 2018-08-11 20:48:06 -07:00
2dd76037be zstd cli can increase level when input is too slow 2018-08-09 15:51:30 -07:00
79a35ac20d minor code comments improvements 2018-08-09 15:16:31 -07:00
2ca7c69167 Fix CDict Attachment to Handle CDicts with Non-Zero Starts
CDicts were previously guaranteed to be generated with `lowLimit=dictLimit=0`.
This is no longer true, and so the old length and index calculations are no
longer valid. This diff fixes them to handle non-zero start indices in CDicts.
2018-08-07 18:14:14 -07:00