52f8c07a3f
Clamp compression level in ZSTD_getCParams_internal() function
2020-11-14 13:26:08 +08:00
f62edf0fe9
Merge pull request #2381 from senhuang42/expand_sequence_extraction_api
...
Add enum to define ZSTD_Sequence type and update sequence extraction API
2020-11-06 13:00:31 -05:00
7d1dea070c
Update unit tests
2020-11-06 11:10:37 -05:00
779df995c6
Implement mergeGeneratedSequences()
2020-11-06 10:55:46 -05:00
51abd58208
Rename getSequences() to generateSequences()
2020-11-06 10:53:22 -05:00
eac309c71b
Add ZSTD_getDictID_fromCDict function to experimental section
2020-11-04 11:37:37 +00:00
f782cac3d4
Change block delimiter removing to linear time approach
2020-11-02 17:06:20 -05:00
3434049c1f
Use ZSTD_memmove() instead of memmove()
2020-11-02 11:43:19 -05:00
d4d0346b40
Update name of enum, clarify documentation
2020-11-02 11:38:17 -05:00
e6178f837f
Revert unnecessary seqCollector adjustment
2020-11-02 10:59:20 -05:00
e8501e00b8
Fix incorrect index increment in merge algorithm
2020-11-02 10:58:41 -05:00
a36fdada57
Add algorithm to remove all delimiters
2020-11-02 10:46:52 -05:00
435a3a0428
Update seqCollector definition
2020-11-02 10:19:26 -05:00
3327932609
Update ZSTD_getSequences function signature
2020-11-02 10:17:59 -05:00
7205e609a9
Merge pull request #2354 from terrelln/stable-buffer
...
Add ZSTD_c_stable{In,Out}Buffer and optimize when set
2020-10-30 15:06:56 -07:00
c37c714ef1
Merge pull request #2376 from senhuang42/clarify_sequence_extraction_api
...
Refine external ZSTD_Sequence API
2020-10-30 15:47:25 -04:00
d4e021fe35
[lib] Avoid allocating the input buffer when ZSTD_c_stableInBuffer is set
...
We don't use it when we have a stable input buffer, so don't allocate
it. I had to slightly modify `ZSTD_copyCCtx()` by storing the
`ZSTD_buffered_policy_e` in the `ZSTD_CCtx`, since `inBuffSize > 0` is
no longer the correct signal for the buffered mode.
2020-10-30 10:55:34 -07:00
24f72789e2
[lib] Skip the input window buffer when ZSTD_c_stableInBuffer is set
...
Compress directly from the `ZSTD_inBuffer`. We still allocate the input
buffer. A following commit will remove that allocation.
2020-10-30 10:55:34 -07:00
fcf81cee5e
[lib] Avoid allocating output buffer when ZSTD_c_stableOutBuffer is set
...
We compress directly to the `ZSTD_outBuffer` so we don't need to
allocate it.
2020-10-30 10:55:34 -07:00
6d5dc93d4e
[lib] Compress directly into output when ZSTD_c_stableOutBuffer is set
...
When we have a stable output buffer always compress directly into the
`ZSTD_outBuffer`. We are allowed to return `dstSizeTooSmall`.
2020-10-30 10:55:34 -07:00
987cb4ca6a
[lib] Take the shortcut when ZSTD_c_stableOutBuffer is set
...
When we have a stable output buffer take the single-pass shortcut.
It is okay to return `dstSizeTooSmall` if the output buffer isn't
big enough, because we know it will never grow.
2020-10-30 10:55:34 -07:00
809b2f2071
[lib] Set ZSTD_c_stable{In,Out}Buffer in ZSTD_compress2()
...
Sets these parameters in ZSTD_compress2() then resets them to their
orignal values after the compression call.
An alternative design could be to add a flush mode `ZSTD_e_singlePass`
which implies `ZSTD_c_stable{In,Out}Buffer` but only for a single
compression call, by directly setting the applied parameters. I've opted
for the smaller change, but this is open for discussion.
2020-10-30 10:55:34 -07:00
c74be3f6de
[lib] Validate buffers when ZSTD_c_stable{In,Out}Buffer is set
...
Adds the validation of the input/output buffers only. They are still
unused.
2020-10-30 10:55:34 -07:00
e3e0775cc8
[API] Add ZSTD_c_stable{In,Out}Buffer parameters
...
This commit adds the parameters and sets the value in the CCtxParams
but it does not do anything with the value.
2020-10-30 10:54:39 -07:00
e2581d9572
[lib] Set appliedParams in zstdmt mode
...
Previously only `nbWorkers` was set. Set all parameters, because that is
what is expected. This is needed for the `ZSTD_c_stable{In,Out}Buffer`
parameters.
2020-10-30 10:54:38 -07:00
536e89c723
Sequence extractor should update CBlockState
2020-10-30 12:13:19 -04:00
32cac2627a
Emit last literals of 0 size as well, to indicate block boundary
2020-10-29 16:41:17 -04:00
69bd5f0654
Correct literalsRead calculation to include longLength
2020-10-29 14:49:37 -04:00
59624f3163
Remove implicit typecast to appease appVeyor windows build
2020-10-28 16:25:09 -04:00
3ed5d053d8
Clarify comments in zstd.h some more
2020-10-28 09:53:09 -04:00
17b700d78a
Merge pull request #2366 from senhuang42/enable_ldm_by_default
...
Enable LDM by default if window size >= 128MB and strategy uses opt parser
2020-10-27 14:59:28 -04:00
3163909d14
Remove unused variable position
2020-10-27 12:58:12 -04:00
dc448563e9
Add test compatibility with last literals in sequences
2020-10-27 12:35:28 -04:00
1d221ecc03
Add support for representing last literals in the extracted seqs
2020-10-27 11:19:48 -04:00
9171f920cd
Improve documentation of seqStore_t
2020-10-27 10:50:22 -04:00
96b0ff7886
Improve documentation regarding various operations in copyBlockSequences
2020-10-27 10:36:06 -04:00
3a11c7eb03
Modify ZSTD_copyBlockSequences to agree with new API
2020-10-27 10:31:40 -04:00
8bdb32aebe
Add a function for LDM enable check
2020-10-20 13:46:02 -04:00
578e889ec1
Move ldm enable to compressStream2()
2020-10-20 13:04:45 -04:00
d28d8a1d72
Include LDM tables size for CCtx size estimation where relevant
2020-10-20 09:21:30 -04:00
b1c7fc5768
Add compatibility for multithreading
2020-10-19 12:07:06 -04:00
590f7f55f0
Add ldm enable condition in ZSTD_resetCCtx_internal
2020-10-19 10:26:17 -04:00
4d01979b62
Expose and call ZSTD_ldm_skipRawSeqStoreBytes()
2020-10-16 20:30:00 -04:00
ee84817fe7
Reset posInSequence when using ZSTD_referenceExternalSequences()
2020-10-14 22:06:08 -04:00
f5d5cd3b40
Merge pull request #2341 from senhuang42/ldm_optimized_for_opt_parser
...
Integrate long distance matches into optimal parser
2020-10-13 13:09:07 -07:00
7e6f91ed84
[minor] Improve docs and add an assert in response to review
2020-10-12 16:43:17 -07:00
d5c688e8ae
Fix ZSTD_adjustCParams_internal() to handle dictionary logic
...
Pass in the `ZSTD_cParamMode_e` to select how we define our cparams.
Based on the mode we either take the `dictSize` into account or we set
it to `0`. See the documentation for `ZSTD_cParamMode_e`.
Some of the modes currently share the same behavior. But they have
distinct modes because they are drastically different cases. E.g.
compression + reprocessing the dictionary and creating a cdict.
Additionally, when downsizing the hashLog and chainLog take the
(adjusted) dictionary size into account, since the size of the
dictionary gets added onto the window size.
Adds a simple test to ensure that we aren't downsizing too far.
2020-10-12 12:50:04 -07:00
fadaab8c7c
[minor improvement] Pass 0 as the content size in the DDS
...
The DDS structure can't be copied into the working tables like the DMS.
So it doesn't need to account for the source size when sizing its
parameters, just the dictionary size.
2020-10-12 12:47:21 -07:00
48ef15fb47
[minor improvement] Pass dictSize when selecting parameters
...
When selecting parameters in streaming compression with a dictionary use
the dictionary size to select the parameters.
2020-10-12 12:47:19 -07:00
012818df99
[refactor] Remove ZSTD_resetCStream_internal()
...
This function is only called in one place. It isn't a logical separation
of duties, and it was only obsfucating the code now, so inline it.
2020-10-12 12:46:10 -07:00
