Yann Collet
2e02cd330d
inform manual users that it's automatically generated
...
suggested by @Eugeny1
2024-10-31 15:06:48 -07:00
Yann Collet
adbb536d00
fixed single file library test
...
copy "zstd_errors.h" alongside "zstd.h" to pass compilation test.
2024-10-31 14:38:36 -07:00
Yann Collet
d9553fd218
elevated ZSTD_getErrorCode() to stable status
...
answering #4183
2024-10-31 14:15:50 -07:00
Yann Collet
15c29168b7
Merge pull request #4180 from facebook/split_param
...
Block splitter control parameter
2024-10-31 14:15:29 -07:00
Yann Collet
bbaba45589
change experimental parameter name
...
from ZSTD_c_useBlockSplitter to ZSTD_c_splitAfterSequences.
2024-10-31 13:43:40 -07:00
Yann Collet
4f93206d62
changed variable name to ZSTD_c_blockSplitterLevel
...
suggested by @terrelln
2024-10-29 11:12:09 -07:00
Yann Collet
f593ccda04
removed trace left over
2024-10-28 16:57:01 -07:00
Yann Collet
fcbf6b014a
fixed minor conversion warning
2024-10-28 16:47:38 -07:00
Yann Collet
37706a677c
added a test
...
test both that the new parameter works as intended,
and that the over-split protection works as intended
2024-10-28 16:31:15 -07:00
Yann Collet
226ae73311
expose new parameter ZSTD_c_blockSplitter_level
2024-10-28 16:31:15 -07:00
Yann Collet
01474bf73b
add internal compression parameter preBlockSplitter_level
...
not yet exposed to the interface.
Also: renames `useBlockSplitter` to `postBlockSplitter`
to better qualify the difference between the 2 settings.
2024-10-28 16:31:15 -07:00
Yann Collet
5bae43b411
Merge pull request #4178 from facebook/split_fromBorders
...
Add fastest block-splitter variant
2024-10-28 16:30:49 -07:00
Yann Collet
4e1a87976f
update regression results
2024-10-25 16:33:00 -07:00
Yann Collet
5b4ce643f0
update ZSTD_splitBlock() documentation
2024-10-25 16:25:02 -07:00
Yann Collet
e557abc8a0
new block splitting variant _fromBorders
...
less precise but still suitable for `fast` strategy.
2024-10-25 16:13:55 -07:00
Yann Collet
7fb5347e88
Merge pull request #4176 from facebook/sample11
...
Added faster block splitter variants for levels 3-7
2024-10-25 16:13:17 -07:00
Yann Collet
c63b5d2294
update regression results
2024-10-24 14:47:55 -07:00
Yann Collet
da2c0dffd8
add faster block splitting heuristic, suitable for dfast strategy
2024-10-24 14:37:00 -07:00
Yann Collet
2366a87ddc
fix minor visual conversion warning
2024-10-24 13:38:12 -07:00
Yann Collet
ea082692aa
added block splitter variant for greedy & lazy (levels 5 to 7)
2024-10-24 13:27:01 -07:00
Yann Collet
326c45bb8e
complete sample11 with reduced fingerprint size
2024-10-24 13:17:56 -07:00
Yann Collet
ca6e55cbf5
reduce splitBlock arguments
2024-10-24 13:17:56 -07:00
Yann Collet
94d7b07425
organize specialization at recordFingerprint level
2024-10-24 13:17:56 -07:00
Yann Collet
566763fdc9
new variant, sampling by 11
2024-10-24 13:17:56 -07:00
Yann Collet
2dddf09056
Merge pull request #4136 from facebook/preSplit
...
Block splitter
2024-10-24 13:17:36 -07:00
Yann Collet
70c77d20d6
update regression results
...
first block is no longer splitted since adding the @savings over-split protection
2024-10-24 11:43:16 -07:00
Yann Collet
90095f056d
apply limit conditions for all splitting strategies
...
instead of just for blind split.
This is in anticipation of adversarial input,
that would intentionally target the sampling pattern of the split detector.
Note that, even without this protection, splitting can never expand beyond ZSTD_COMPRESSBOUND(),
because this upper limit uses a 1KB block size worst case scenario,
and splitting never creates blocks thath small.
The protection is more to ensure that data is not expanded by more than 3-bytes per 128 KB full block,
which is a much stricter limit.
2024-10-24 11:36:56 -07:00
Yann Collet
bbda1acf85
update regression results
2024-10-23 15:56:56 -07:00
Yann Collet
c80645a055
stricter limits to ensure expansion factor with blind-split strategy
...
issue reported by @terrelln
2024-10-23 14:55:10 -07:00
Yann Collet
7d3e5e3ba1
split all full 128 KB blocks
...
this helps make the streaming behavior more consistent,
since it does no longer depend on having more data presented on the input.
suggested by @terrelln
2024-10-23 14:18:48 -07:00
Yann Collet
b68ddce818
rewrite fingerprint storage to no longer need 64-bit members
...
so that it can be stored using standard alignment requirement (sizeof(void*)).
Distance function still requires 64-bit signed multiplication though,
so it won't change the issue regarding the bug in ubsan for clang 32-bit on github ci.
2024-10-23 11:50:57 -07:00
Yann Collet
57239c4d3b
fixed minor strict pedantic C90 issue
2024-10-23 11:50:57 -07:00
Yann Collet
18b1e67223
fixed extraneous return
...
strict C90 compliance test
2024-10-23 11:50:57 -07:00
Yann Collet
d2eeed53dc
updated compression results
...
due to integration of `sample5` strategy, leading to better compression ratios on a range of levels
2024-10-23 11:50:57 -07:00
Yann Collet
0be334d208
fixes static state allocation check
...
detected by @felixhandte
2024-10-23 11:50:57 -07:00
Yann Collet
06b7cfabf8
rewrote ZSTD_cwksp_initialAllocStart() to be easier to read
...
following a discussion with @felixhandte
2024-10-23 11:50:57 -07:00
Yann Collet
16450d0732
rewrite penalty update
...
suggested by @terrelln
2024-10-23 11:50:57 -07:00
Yann Collet
1ec5f9f1f6
changed loop exit condition so that there is no need to assert() within the loop.
2024-10-23 11:50:57 -07:00
Yann Collet
4662f6e646
renamed: FingerPrint => Fingerprint
...
suggested by @terrelln
2024-10-23 11:50:57 -07:00
Yann Collet
ea85dc7af6
conservatively estimate over-splitting in presence of incompressible loss
...
ensure data can never be expanded by more than 3 bytes per full block.
2024-10-23 11:50:57 -07:00
Yann Collet
5ae34e4c96
ensure lastBlock is correctly determined
...
reported by @terrelln
2024-10-23 11:50:57 -07:00
Yann Collet
7bad787d8b
made ZSTD_isPower2() an inline function
2024-10-23 11:50:57 -07:00
Yann Collet
a167571db5
added a faster block splitter variant
...
that samples 1 in 5 positions.
This variant is fast enough for lazy2 and btlazy2,
but it's less good in combination with post-splitter at higher levels (>= btopt).
2024-10-23 11:50:57 -07:00
Yann Collet
1c62e714ab
minor split optimization
...
let's fill the initial stats directly into target fingerprint
2024-10-23 11:50:57 -07:00
Yann Collet
dac26eaeac
updated regression test results
2024-10-23 11:50:57 -07:00
Yann Collet
4ce91cbf2b
fixed workspace alignment on non 64-bit systems
2024-10-23 11:50:57 -07:00
Yann Collet
cae8d13294
splitter workspace is now provided by ZSTD_CCtx*
2024-10-23 11:50:56 -07:00
Yann Collet
4685eafa81
fix alignment test
...
for non 64-bit systems
2024-10-23 11:50:56 -07:00
Yann Collet
433f4598ad
fixed minor conversion warnings on Visual
2024-10-23 11:50:56 -07:00
Yann Collet
73a6653653
ZSTD_splitBlock_4k() uses externally provided workspace
...
ideally, this workspace would be provided from the ZSTD_CCtx* state
2024-10-23 11:50:56 -07:00
