58476bcf7f
Don't shrink window log in ZSTD_getCParams()
...
Treat ZSTD_getCParams() and ZSTD_adjustCParams() in the same way
we treat streaming compression. Choose parameters based on the
dictionary size + source size, and assume the source size is small
if unkown. But, don't shrink the window log down in
ZSTD_adjustCParams_internal().
2021-01-04 15:54:09 -08:00
9d31c704d5
Don't shrink window log when streaming with a dictionary
...
Fixes #2442 .
1. When creating a dictionary keep the same behavior as before.
Assume the source size is 513 bytes when adjusting parameters.
2. When calling ZSTD_getCParams() or ZSTD_adjustCParams() keep
the same behavior as before.
3. When attaching a dictionary keep the same behavior of ignoring
the dictionary size. When streaming this will select the
largest parameters and not adjust them down. But, the CDict
will use the correctly sized parameters, which seems like the
right tradeoff.
4. When not attaching a dictionary (either forced not to, or
using a prefix dictionary) we select parameters based on the
dictionary size + source size, and assume the source size is
small, which is the same behavior as before. But, now we don't
adjust the window log (and hash and chain log) down when the
source size is unknown.
When the source size is unknown all cdicts should attach, except
when the user disables attaching, or `forceWindow` is used. This
means that when streaming with a CDict we end up in the good case
where we get small CDict parameters, and large source parameters.
TODO: Add a streaming + dictionary regression test case.
2021-01-04 15:54:09 -08:00
66e811d782
[license] Update year to 2021
2021-01-04 17:53:52 -05:00
5c41490bfe
Use pre-defined constants
2020-12-21 11:52:05 -05:00
7e11bd012b
Implement skippable frame function
2020-12-21 11:13:22 -05:00
0b39531d75
moving all references to release branch
...
was previously `master`
2020-12-16 23:00:35 -08:00
9dab03db90
Create Enum to Represent Static/Dynamic Allocation Distinction in cwksp
2020-12-09 14:57:37 -05:00
db9e73cb07
Don't ASAN-Poison Statically-Allocated Workspaces
...
Addresses #2286 .
2020-12-09 13:00:47 -05:00
c238db046f
Merge pull request #2414 from terrelln/mt-progress
...
[lib] Ensure that multithreaded compression always makes some progress
2020-12-04 16:30:08 -08:00
4c58cb8383
[lib] Ensure that multithreaded compression always makes some progress
2020-12-03 20:25:14 -08:00
6672689e7e
Merge pull request #2406 from terrelln/linux-wrapper-api
...
[linux] Add the linux wrapper API
2020-12-02 16:49:03 -08:00
894ae36675
Merge pull request #2390 from animalize/clamp_level
...
Clamp compression level
2020-12-02 14:35:58 -08:00
2cbd038528
Move max nb seq check to per-block
2020-12-02 12:11:32 -05:00
3cda5fae77
[minor][lib] Remove double semicolon
2020-12-02 01:08:08 -08:00
3efe9c902b
Add sequence nb validation to compressSequences(), adjust minMatch comparisons
2020-12-01 10:54:45 -05:00
4c5f337248
Use cctx's minMatch instead of global MINMATCH, make fuzzer use validation
2020-11-30 15:41:20 -05:00
c5fbd55dac
Merge pull request #2387 from senhuang42/compress_sequence_API
...
[RFC] New sequence compression API
2020-11-20 16:54:20 -05:00
7742f076b4
Add experimental param for sequence validation
2020-11-20 11:57:41 -05:00
0e32928b7d
Remove unnecessary repcode backup, apply style choices, use function pointer
2020-11-20 11:02:19 -05:00
e924a0fa51
Explicit cast for visual warnings
...
Github has automatic commits now! Cool
Co-authored-by: Nick Terrell <nickrterrell@gmail.com >
2020-11-19 17:32:40 -05:00
dcbbf7c09f
Unroll isRLE loop
2020-11-19 12:38:13 -05:00
05c0229668
Clean up visual conversion warnings
2020-11-18 15:36:29 -05:00
d6d7ba2a1f
Modification to offset validation to include entire sequence
2020-11-17 10:13:22 -05:00
8f3136a9c7
Fix assert edge case, improve documentation in zstd.h
2020-11-16 18:05:35 -05:00
f6baad87d6
Fix warnings and make validation enabled by default
2020-11-16 12:00:06 -05:00
55b90ef010
Fix unit tests to agree with new changes
2020-11-16 11:36:37 -05:00
7f563b0519
Add new sequence format as an experimental CCtx param
2020-11-16 10:49:17 -05:00
347824ad73
Overhaul logic to simplify, add in proper validations, fix match splitting
2020-11-16 10:49:17 -05:00
46824cb018
Add new sequence compress api params to cctx
2020-11-16 10:49:17 -05:00
48405b4633
Fix srcSize=0 edge case
2020-11-16 10:49:17 -05:00
022e6d81e7
Fix literals length calculation
2020-11-16 10:49:17 -05:00
dad20b5ccb
Remove dstCapacity error check
2020-11-16 10:49:17 -05:00
b8e16a2057
Remove extraneous function in this API
2020-11-16 10:49:17 -05:00
f29507c4fc
Add check comparing offset to window size
2020-11-16 10:49:17 -05:00
7a6e46a92f
Fix MSAN errors
2020-11-16 10:49:17 -05:00
cc2642bd17
Address edge case with endPosInSequence
2020-11-16 10:49:17 -05:00
fd10007174
Change debug levels to appropriate ones
2020-11-16 10:49:17 -05:00
2db8441245
Add RLE support
2020-11-16 10:49:17 -05:00
dfef298336
Fix various build warnings
2020-11-16 10:49:17 -05:00
2bbdddf24e
Add test case to roundtrip using ZSTD_getSequences() and ZSTD_compressSequences()
2020-11-16 10:49:16 -05:00
5fd69f8173
Add documentation for new api functions
2020-11-16 10:49:16 -05:00
e8b7fdb64b
Refactor for enhanced code clarity
2020-11-16 10:49:16 -05:00
c675fb46f1
Rename internal function compressSequences(), and promote new *_ext() functions to their actual name
2020-11-16 10:49:16 -05:00
013434e1e4
Add another API function to compress with existing CCTX
2020-11-16 10:49:16 -05:00
c44ce29013
More adjustments to improve code clarity
2020-11-16 10:49:16 -05:00
48f67da854
Pull compressStream2() transparent initialization into its own function
2020-11-16 10:49:16 -05:00
c86151f53c
Add initial support for new ZSTD_Sequence mode
2020-11-16 10:49:16 -05:00
e0f26afce9
Add sequence compression format param
2020-11-16 10:49:16 -05:00
f51af9a609
Always ensure sequenceRange updates properly, add more error forwarding
2020-11-16 10:49:16 -05:00
1a449688fd
Various minor logical refactors to improve clarity
2020-11-16 10:49:16 -05:00
