1
0
mirror of https://github.com/facebook/zstd.git synced 2025-07-05 23:27:28 +02:00
Commit Graph

219 Commits

Author SHA1 Message Date
d6770f80af minor : rewrite unit tests using CHECK_Z macro 2017-09-28 02:14:48 -07:00
9fe50ed623 fixed maximum windowLog for zstreamtest --newapi
for compatibility with low memory VM
2017-09-28 01:42:06 -07:00
9b5b47ac93 ensure adjustCParams adjust hLog and cLog even without srcSize
It would previously exit when srcSize is unknown.
But in the case of custom parameters,
hLog and cLog can still be too large in comparison with windowLog.

Reduces maximum memory allocated during zstreamtest --newapi
2017-09-28 01:25:40 -07:00
aa800c4793 reduced memory usage of zstreamtest --newapi
to run on memory-constrained VM
2017-09-27 18:00:15 -07:00
bc32b40b98 reduced zstreamtest --mt memory load
adjust compression level, hence memory usage, depending on nb threads
in order to run correctly on memory-starved VM.
2017-09-27 17:27:38 -07:00
54a827fff0 Merge branch 'dev' into newFormats
Fixed conflicts in zstdmt_compress.c
2017-09-27 16:39:40 -07:00
bfabd1d4dc fixed zstreamtest decoding error
same error (wrong output buffer size) was present
on --mt and --new_api tests.
2017-09-27 01:01:11 -07:00
ca306c1c84 fixed a bug in zstreamtest
decoder output buffer would receive a wrong size.

In previous version, ZSTD_decompressStream() would blindly trust the caller that pos <= size.
In this version, this condition is actively checked,
and the function returns an error code if this condition is not respected.

This check could also be done with an assert(),
but since this is a user-facing interface, it seems better to keep this check at runtime.
2017-09-27 00:39:41 -07:00
c233bdbaee Increase maximum window size
* Maximum window size in 32-bit mode is 1GB, since allocations for 2GB fail
  on my Mac.
* Maximum window size in 64-bit mode is 2GB, since that is the largest
  power of 2 that works with the overflow prevention.
* Allow `--long=windowLog` to set the window log, along with
  `--zstd=wlog=#`. These options also set the window size during
  decompression, but don't override `--memory=#` if it is set.
* Present a helpful error message when the window size is too large during
  decompression.
* The long range matcher defaults to a hash log 7 less than the window log,
  which keeps it at 20 for window log 27.
* Keep the default long range matcher window size and the default maximum
  window size at 27 for the API and CLI.
* Add tests that use the maximum window size and hash size for compression
  and decompression.
2017-09-26 14:00:01 -07:00
96f0cde31a minor function rename
ZSTD_estimateCStreamSize_advanced_usingCParams -> ZSTD_estimateCStreamSize_usingCParams
_usingX is clear.
_advanced feels redundant
2017-09-24 16:47:02 -07:00
f1571dad8f Merge pull request #838 from stellamplau/ldm-mergeDev
Add long distance matcher
2017-09-13 13:24:08 -07:00
3d8e313f64 Reduce ldm hash table size in test 2017-09-11 17:21:28 -07:00
eb3327c10a Merge branch 'dev' of https://github.com/facebook/zstd into ldm-mergeDev 2017-09-11 15:00:01 -07:00
b3f33ccfb3 use ZSTD_decodingBufferSize_min() inside ZSTD_decompressStream()
Use same definition as public one
minor : reduce allocated buffer size in some cases
(when frameContentSize is known and == windowSize)
2017-09-09 14:37:28 -07:00
3128e03be6 updated license header
to clarify dual-license meaning as "or"
2017-09-08 00:09:23 -07:00
9e4060200b Add tests and fix pointer alignment 2017-09-06 09:14:05 -07:00
67d4a6161c Add ldmBucketSizeLog param 2017-09-02 21:55:29 -07:00
a1f04d518d Move hashEveryLog to cctxParams and update cli 2017-09-01 15:05:47 -07:00
767a0b3be1 Move ldm hashLog, bucketLog, and mml to cctxParams 2017-09-01 12:24:59 -07:00
17d8e0bdcc Merge remote-tracking branch 'upstream/longRangeMatcher' into ldm-integrate 2017-09-01 10:19:38 -07:00
8081becadc Add long distance matching as a CCtxParam 2017-09-01 09:18:58 -07:00
d7ad99b2ab Merge branch 'longRangeMatcher' into dev 2017-08-31 18:08:37 -07:00
6a546efb8c Add long distance matcher
Move last literals section to ZSTD_block_internal
2017-08-31 12:53:19 -07:00
90a31bfa16 Pass dictMode to ZSTDMT_initCStream; fix nits
- Return error code in estimate{CCtx,CStream}Size functions
2017-08-30 16:19:07 -07:00
82d636b76a Rename applyCCtxParams() 2017-08-29 18:03:06 -07:00
c88fb9267f Replace 'byReference' with enum 2017-08-29 11:55:02 -07:00
b5b9275e67 Rename estimateCCtxSize_advanced() and estimateCStreamSize_advanced() 2017-08-29 10:49:29 -07:00
eb7bbab36a Remove ZSTD_p_refDictContent and dictContentByRef 2017-08-25 11:11:45 -07:00
6f1a21c7e9 Remove formatting-only changes 2017-08-23 10:24:19 -07:00
73c73bf16a Reduce code duplication in zstreamtest 2017-08-21 12:41:19 -07:00
91b30dbe84 Remove test parameter 2017-08-21 10:09:06 -07:00
f181f33bdf Disable tests and refactor 2017-08-21 01:59:08 -07:00
023b24e6d4 Add cctx param tests 2017-08-20 22:55:07 -07:00
32fb407c9d updated a bunch of headers
for the new license
2017-08-18 16:52:05 -07:00
052a95f77c fix : ZSTDMT_compress_advanced() correctly generates checksum
when params.fParams.checksumFlag==1.
This use case used to be impossible when only ZSTD_compress() was available
2017-07-11 17:18:26 -07:00
4616fad18b improved ZSTDMT_compress() memory usage
does not need the input buffer for streaming operations

also : reduced a few tests time length
2017-07-10 17:16:41 -07:00
afb0aca739 zstreamtest : big tests are only enabled in 64-bits mode
to avoid requesting too much memory in 32-bits mode during MT tests
2017-06-29 18:19:09 -07:00
62f7efc04a added a test to ensure ZSTD_refPrefix() doesn't impact the following compression job 2017-06-28 16:25:13 -07:00
33a6639039 fixed ZSTD_refPrefix with Multithread-enabled CCtx 2017-06-28 11:09:43 -07:00
1fd2df9c2c fixed zstreamtest
modifying compression parameters is not allowed after ZSTD_CCtx_loadDictionary()
but it is still allowed after ZSTD_refPrefix()
2017-06-28 09:57:59 -07:00
2e4274262d controlled dictMode 2017-06-27 17:09:12 -07:00
b7372933b8 implemented ZSTD_refPrefix() 2017-06-27 15:49:12 -07:00
fecc721fd9 added parameter ZSTD_p_refDictContent 2017-06-27 11:46:39 -07:00
dde10b23fe refactored ZSTD_estimateDStreamSize()
now uses windowSize as argument.
Also : created ZSTD_estimateDStreamSize_fromFrame()
2017-06-26 17:44:26 -07:00
09ae03a570 ZSTD_estimateCDictSize_advanced()
ZSTD_estimateCDictSize() now uses same arguments as ZSTD_createCDict()
ZSTD_estimateCDictSize_advanced() uses same arguments as ZSTD_createCDict_advanced()
2017-06-26 16:47:32 -07:00
0c9a915a28 ZSTD_estimateCStreamSize_advanced() 2017-06-26 16:02:25 -07:00
cca1ed14db Merge branch 'advancedAPI2' of github.com:facebook/zstd into advancedAPI2 2017-06-23 00:13:03 -07:00
32f546c763 zstreamtest : disabled multi-threading tests with --newapi --no-big-tests
--no-big-tests is typically used in combination with qemu-user-static
qemu-user-static allocated 4 GB of RAM upfront.
On 2 GB VM, this can degenerate into a crash.
It's not a problem as long as memory is not used.
But with multi-threading enabled, memory fragmentation kicks in,
so the amoung of RAM effectively touched increases,
and can pass beyond the 2 GB limit of the VM.

In single-threaded mode, there is no such issue :
memory requirement is smaller, and remains well-located,
so very little fragmentation is expected.

This modification should make `qemu-arm-static zstreamtests --newapi --no-big-tests`
work fine on Travis CI.
2017-06-23 00:09:02 -07:00
f99c2c1a22 reduced --no-big-tests even more
to pass tests on qemu-aarch64
2017-06-21 23:35:58 -07:00
49f8459d11 fixed minor cast warning 2017-06-21 18:43:39 -07:00