krak/zstd
krak/zstd
1
0
Fork 0
mirror of https://github.com/facebook/zstd.git synced 2025-03-07 09:26:03 +02:00
Code Issues Releases 1 Activity
8,296 Commits 25 Branches 76 Tags
Commit Graph

294 Commits

Author SHA1 Message Date
senhuang42
9d936d61d2 Reduce number of memcpy() calls 2020-11-13 19:43:30 -05:00
senhuang42
1a8af0de73 Improve unit test 2020-11-12 11:09:09 -05:00
sen
f62edf0fe9
Merge pull request #2381 from senhuang42/expand_sequence_extraction_api 
Add enum to define ZSTD_Sequence type and update sequence extraction API
 2020-11-06 13:00:31 -05:00
senhuang42
7d1dea070c Update unit tests 2020-11-06 11:10:37 -05:00
senhuang42
51abd58208 Rename getSequences() to generateSequences() 2020-11-06 10:53:22 -05:00
Luke Pitt
eac309c71b Add ZSTD_getDictID_fromCDict function to experimental section 2020-11-04 11:37:37 +00:00
senhuang42
c54a25b666 Revert compressibility change 2020-11-02 11:38:58 -05:00
senhuang42
d4d0346b40 Update name of enum, clarify documentation 2020-11-02 11:38:17 -05:00
senhuang42
9102f30dbf Update unit test 2020-11-02 11:30:31 -05:00
senhuang42
3327932609 Update ZSTD_getSequences function signature 2020-11-02 10:17:59 -05:00
sen
c37c714ef1
Merge pull request #2376 from senhuang42/clarify_sequence_extraction_api 
Refine external ZSTD_Sequence API
 2020-10-30 15:47:25 -04:00
sen
ff93440fc6
Merge pull request #2375 from senhuang42/ldm_oss_fuzz_testcase 
Add a test case for LDM + opt parser with small uncompressible block
 2020-10-29 09:32:05 -04:00
senhuang42
7198ebb213 Un-mix declarations and code 2020-10-28 18:51:03 -04:00
senhuang42
60a52c29e6 Add check for allocation 2020-10-28 16:22:22 -04:00
Nick Terrell
599ff58e08
Merge pull request #2339 from terrelln/zstdmt-stability 
Fix zstdmt stability issues and clean up the zstdmt code
 2020-10-27 19:43:13 -07:00
senhuang42
169fc07aa1 Move test to appropriate location 2020-10-27 16:59:43 -04:00
senhuang42
db0b5d7d1e Add test to fuzzer.c 2020-10-27 16:57:24 -04:00
sen
17b700d78a
Merge pull request #2366 from senhuang42/enable_ldm_by_default 
Enable LDM by default if window size >= 128MB and strategy uses opt parser
 2020-10-27 14:59:28 -04:00
senhuang42
dc448563e9 Add test compatibility with last literals in sequences 2020-10-27 12:35:28 -04:00
senhuang42
81a2c02d8f Move ldm no regression test to fuzzer longtests 2020-10-19 15:28:46 -04:00
senhuang42
df470e176b Add unit test for no cctx requested params change 2020-10-19 10:52:41 -04:00
Yann Collet
f5d5cd3b40
Merge pull request #2341 from senhuang42/ldm_optimized_for_opt_parser 
Integrate long distance matches into optimal parser
 2020-10-13 13:09:07 -07:00
Nick Terrell
c51a9e79b9 [zstdmt] Rip out the zstdmt API 
This commit leaves only the functions used by zstd_compress.c. All other
functions have been removed from the API. The ZSTDMT unit tests in
fuzzer.c and zstreamtest.c have been rewritten to use the ZSTD API. And
the --mt zstreamtest tests have been ripped out.
 2020-10-12 12:55:16 -07:00
Nick Terrell
d5c688e8ae Fix ZSTD_adjustCParams_internal() to handle dictionary logic 
Pass in the `ZSTD_cParamMode_e` to select how we define our cparams.
Based on the mode we either take the `dictSize` into account or we set
it to `0`. See the documentation for `ZSTD_cParamMode_e`.

Some of the modes currently share the same behavior. But they have
distinct modes because they are drastically different cases. E.g.
compression + reprocessing the dictionary and creating a cdict.

Additionally, when downsizing the hashLog and chainLog take the
(adjusted) dictionary size into account, since the size of the
dictionary gets added onto the window size.

Adds a simple test to ensure that we aren't downsizing too far.
 2020-10-12 12:50:04 -07:00
Nick Terrell
7083f79008 [bug] Fix dictContentType when reprocessing cdict 
Conditions to trigger:
* CDict is loaded as raw content.
* CDict starts with the zstd dictionary magic number.
* The CDict is reprocessed (not attached or copied).
* The new API is used (streaming or `ZSTD_compress2()`).

Bug: The dictionary is loaded as a zstd dictionary, not a raw content
dictionary, because the dict content type is set to `ZSTD_dct_auto`.

Fix: Pass in the dictionary content type from cdict creation to the call
to `ZSTD_compress_insertDictionary()`.

Test: Added a test case that exposes the bug, and fixed the raw
content tests to not modify the `dictBuffer`, which makes all future
tests with the `dictBuffer` raw content, which doesn't seem intentional.
 2020-10-12 12:46:10 -07:00
senhuang42
e96ea5d147 Fix static analyze fuzzer.c error 2020-10-07 13:56:25 -04:00
senhuang42
b8bfc4e63d Add cSize regression test to fuzzer.c 2020-10-07 13:56:25 -04:00
Nick Terrell
2e7d174130 Reset all decompression parameters in ZSTD_DCtx_reset() 
* Reset all decompression parameters in `ZSTD_DCtx_reset()` when
  resetting parameters.
* Add a test case.
 2020-10-01 14:19:21 -07:00
W. Felix Handte
9398acb245 Move Last Two Long Tests in fuzzer.c into Separate --long-tests Section 2020-09-17 13:31:10 -04:00
Yann Collet
dec1a78d3e minor fix casting for Visual 2020-09-14 11:46:23 -07:00
Yann Collet
c91a0855f8 check endDirective in ZSTD_compressStream2() 
fix #2297
also :
- `assert()` `endDirective` in `ZSTD_compressStream_internal()`, for debug mode
- add relevant tests
 2020-09-14 10:56:08 -07:00
W. Felix Handte
d6246d4a0f Print More During Fuzzer Test to Avoid CI Killing it Due to Timeout 
This is kind of hacky. And maybe this test doesn't need to be permanently as
exhaustive as it is now. But while we're actively developing the DDSS, we
should ensure it's compatible across many different modes.
 2020-09-10 23:35:42 -04:00
W. Felix Handte
6d3f816b3e Test Fewer Dictionary Sizes 2020-09-10 22:30:52 -04:00
W. Felix Handte
b6df3fd438 Fix Debug Logging in 32-bit Build 2020-09-10 22:10:02 -04:00
W. Felix Handte
2cc2b40a1b Test DDSS A Little More Thoroughly 2020-09-10 22:10:02 -04:00
W. Felix Handte
b81f3a37f9 Easy: Fix Test 2020-09-10 18:51:52 -04:00
W. Felix Handte
2cf6cfc55f Add Fuzzer Test for the Various Dict Attachment Strategies 2020-09-10 18:51:52 -04:00
Yann Collet
f82d9865b9
Merge pull request #2278 from senhuang42/ignore_checksum_advanced_param 
New advanced decompression param to ignore checksums
 2020-08-25 12:08:53 -07:00
Nick Terrell
614e446000
Merge pull request #2271 from terrelln/small-blocks 
Small block optimizations
 2020-08-24 18:54:33 -07:00
senhuang42
a030560d62 Add new DCtx param: validateChecksum and update unit tests 2020-08-24 17:28:00 -04:00
Nick Terrell
1302f8d676 [fix] Always return dstSize_tooSmall when it is the case 2020-08-24 13:38:13 -07:00
senhuang42
44c54a3e31 Addressing comments: more comments, cleanup, remove extra function, checksum logic 2020-08-24 16:14:19 -04:00
senhuang42
ffaa0df76d Document change in CLI for --no-check during decompression in --help menu 2020-08-24 09:49:12 -04:00
senhuang42
20eb095882 Added unit test to fuzzer.c, changed definition param name 2020-08-22 13:26:33 -04:00
senhuang42
1b34b15e6b Adding CLI capability to invoke decompression with no checksum 2020-08-21 17:49:30 -04:00
senhuang42
6a8dbdcd1f Modify decompression loop to gnore checksums if flag is enabled 2020-08-21 16:46:46 -04:00
Nick Terrell
575731b6db Use ncount=1 when < 4096 symbols 2020-08-18 16:47:53 -07:00
Nick Terrell
651d3d73e0 [test] Update the ldm loadedDictEnd test to cover zstdmt 2020-05-19 16:14:14 -07:00
Nick Terrell
7b317b4876 [test] Test that the ldm dictionary gets invalidated on reset 2020-05-18 16:00:28 -07:00
Nick Terrell
87dbd6d4bf [test] Improve LDM forceMaxWindow test 2020-05-18 15:11:18 -07:00