-
krak released this
2024-03-22 00:05:51 +02:00 | 564 commits to dev since this releaseThis release highlights the deployment of Google Chrome 123, introducing zstd-encoding for Web traffic, introduced as a preferable option for compression of dynamic contents. With limited web server support for zstd-encoding due to its novelty, we are launching an updated Zstandard version to facilitate broader adoption.
New stable parameter ZSTD_c_targetCBlockSize
Using zstd compression for large documents over the Internet, data is segmented into smaller blocks of up to 128 KB, for incremental updates. This is crucial for applications like Chrome that process parts of documents as they arrive. However, on slow or congested networks, there can be some brief unresponsiveness in the middle of a block transmission, delaying update. To mitigate such scenarios, libzstd introduces the new parameter ZSTD_c_targetCBlockSize, enabling the division of blocks into even smaller segments to enhance initial byte delivery speed. Activating this feature incurs a cost, both runtime (equivalent to -2% speed at level 8) and a slight compression efficiency decrease (<0.1%), but offers some interesting latency reduction, notably beneficial in areas with less powerful network infrastructure.Granular binary size selection
libzstd provides build customization, including options to compile only the compression or decompression modules, minimizing binary size. Enhanced in v1.5.6 (source), it now allows for even finer control by enabling selective inclusion or exclusion of specific components within these modules. This advancement aids applications needing precise binary size management.Miscellaneous Enhancements
This release includes various minor enhancements and bug fixes to enhance user experience. Key updates include an expanded list of recognized compressed file suffixes for the --exclude-compressed flag, improving efficiency by skipping presumed incompressible content. Furthermore, compatibility has been broadened to include additional chipsets (sparc64, ARM64EC, risc-v) and operating systems (QNX, AIX, Solaris, HP-UX).Change Log
api: Promote ZSTD_c_targetCBlockSize to Stable API by @felixhandte
api: new experimental ZSTD_d_maxBlockSize parameter, to reduce streaming decompression memory, by @terrelln
perf: improve performance of param ZSTD_c_targetCBlockSize, by @Cyan4973
perf: improved compression of arrays of integers at high compression, by @Cyan4973
lib: reduce binary size with selective built-time exclusion, by @felixhandte
lib: improved huffman speed on small data and linux kernel, by @terrelln
lib: accept dictionaries with partial literal tables, by @terrelln
lib: fix CCtx size estimation with external sequence producer, by @embg
lib: fix corner case decoder behaviors, by @Cyan4973 and @aimuz
lib: fix zdict prototype mismatch in static_only mode, by @ldv-alt
lib: fix several bugs in magicless-format decoding, by @embg
cli: add common compressed file types to --exclude-compressed by @daniellerozenblit (requested by @dcog989)
cli: fix mixing -c and -o commands with --rm, by @Cyan4973
cli: fix erroneous exclusion of hidden files with --output-dir-mirror by @felixhandte
cli: improved time accuracy on BSD, by @felixhandte
cli: better errors on argument parsing, by @KapJI
tests: better compatibility with older versions of grep, by @Cyan4973
tests: lorem ipsum generator as default content generator, by @Cyan4973
build: cmake improvements by @terrelln, @sighingnow, @gjasny, @JohanMabille, @Saverio976, @gruenich, @teo-tsirpanis
build: bazel support, by @jondo2010
build: fix cross-compiling for AArch64 with lld by @jcelerier
build: fix Apple platform compatibility, by @nidhijaju
build: fix Visual 2012 and lower compatibility, by @Cyan4973
build: improve win32 support, by @DimitriPapadopoulos
build: better C90 compliance for zlibWrapper, by @emaste
port: make: fat binaries on macos, by @mredig
port: ARM64EC compatibility for Windows, by @dunhor
port: QNX support by @klausholstjacobsen
port: MSYS2 and Cygwin makefile installation and test support, by @QBos07
port: risc-v support validation in CI, by @Cyan4973
port: sparc64 support validation in CI, by @Cyan4973
port: AIX compatibility, by @likema
port: HP-UX compatibility, by @likema
doc: Improved specification accuracy, by @elasota
bug: Fix and deprecate ZSTD_generateSequences (#3981), by @terrellnFull change list (auto-generated)
Add win32 to windows-artifacts.yml by @Kim-SSi in #3600
Fix mmap-dict help output by @daniellerozenblit in #3601
[oss-fuzz] Fix simple_round_trip fuzzer with overlapping decompression by @terrelln in #3612
Reduce streaming decompression memory by (128KB - blockSizeMax) by @terrelln in #3616
removed travis & appveyor scripts by @Cyan4973 in #3621
Add ZSTD_d_maxBlockSize parameter by @terrelln in #3617
[doc] add decoder errata paragraph by @Cyan4973 in #3620
add makefile entry to build fat binary on macos by @mredig in #3614
Disable unused variable warning in msan configurations by @danlark1 in #3624
#3634
Allow Build-Time Exclusion of Individual Compression Strategies by @felixhandte in #3623
Get zstd working with ARM64EC on Windows by @dunhor in #3636
minor : update streaming_compression example by @Cyan4973 in #3631
Fix UBSAN issue (zero addition to NULL) by @terrelln in #3658
Add options in Makefile to cmake by @sighingnow in #3657
fix a minor inefficiency in compress_superblock by @Cyan4973 in #3668
Fixed a bug in the educational decoder by @Cyan4973 in #3659
changed LLU suffix into ULL for Visual 2012 and lower by @Cyan4973 in #3664
fixed decoder behavior when nbSeqs==0 is encoded using 2 bytes by @Cyan4973 in #3669
detect extraneous bytes in the Sequences section by @Cyan4973 in #3674
Bitstream produces only zeroes after an overflow event by @Cyan4973 in #3676
Update FreeBSD CI images to latest supported releases by @emaste in #3684
Clean up a false error message in the LDM debug log by @embg in #3686
Hide ASM symbols on Apple platforms by @nidhijaju in #3688
Changed the decoding loop to detect more invalid cases of corruption sooner by @Cyan4973 in #3677
Fix Intel Xcode builds with assembly by @gjasny in #3665
Save one byte on the frame epilogue by @Coder-256 in #3700
Update fileio.c: fix build failure with enabled LTO by @LocutusOfBorg in #3695
fileio_asyncio: handle malloc fails in AIO_ReadPool_create by @void0red in #3704
Fix typographical error in README.md by @nikohoffren in #3701
Fixed typo by @alexsifivetw in #3712
Improve dual license wording in README by @terrelln in #3718
Unpoison Workspace Memory Before Custom-Free by @felixhandte in #3725
added ZSTD_decompressDCtx() benchmark option to fullbench by @Cyan4973 in #3726
No longer reject dictionaries with literals maxSymbolValue < 255 by @terrelln in #3731
fix: ZSTD_BUILD_DECOMPRESSION message by @0o001 in #3728
Updated Makefiles for full MSYS2 and Cygwin installation and testing … by @QBos07 in #3720
Work around nullptr-with-nonzero-offset warning by @terrelln in #3738
Fix & refactor Huffman repeat tables for dictionaries by @terrelln in #3737
zdictlib: fix prototype mismatch by @ldv-alt in #3733
Fixed zstd cmake shared build on windows by @JohanMabille in #3739
Added qnx in the posix test section of platform.h by @klausholstjacobsen in #3745
added some documentation on ZSTD_estimate*Size() variants by @Cyan4973 in #3755
Improve macro guards for ZSTD_assertValidSequence by @terrelln in #3770
Stop suppressing pointer-overflow UBSAN errors by @terrelln in #3776
fix x32 tests on Github CI by @Cyan4973 in #3777
Fix new typos found by codespell by @DimitriPapadopoulos in #3771
Do not test WIN32, instead test _WIN32 by @DimitriPapadopoulos in #3772
Fix a very small formatting typo in the lib/README.md file by @dloidolt in #3763
Fix pzstd Makefile to allow setting DESTDIR and BINDIR separately by @paulmenzel in #3752
Remove FlexArray pattern from ZSTDMT by @Cyan4973 in #3786
solving flexArray issue #3785 in fse by @Cyan4973 in #3789
Add doc on how to use it with cmake FetchContent by @Saverio976 in #3795
Correct FSE probability bit consumption in specification by @elasota in #3806
Add Bazel module instructions to README.md by @jondo2010 in #3812
Clarify that a stream containing too many Huffman weights is invalid by @elasota in #3813
[cmake] Require CMake version 3.5 or newer by @gruenich in #3807
Three fixes for the Linux kernel by @terrelln in #3822
[huf] Improve fast huffman decoding speed in linux kernel by @terrelln in #3826
[huf] Improve fast C & ASM performance on small data by @terrelln in #3827
update xxhash library to v0.8.2 by @Cyan4973 in #3820
Modernize macros to use do { } while (0) by @terrelln in #3831
Clarify that the presence of weight value 1 is required, and a lone implied 1 weight is invalid by @elasota in #3814
Move offload API params into ZSTD_CCtx_params by @embg in #3839
Update FreeBSD CI: drop 12.4 (nearly EOL) by @emaste in #3845
Make offload API compatible with static CCtx by @embg in #3854
zlibWrapper: convert to C89 / ANSI C by @emaste in #3846
Fix a nullptr dereference in ZSTD_createCDict_advanced2() by @michoecho in #3847
Cirrus-CI: Add FreeBSD 14 by @emaste in #3855
CI: meson: use builtin handling for MSVC by @eli-schwartz in #3858
cli: better errors on argument parsing by @KapJI in #3850
Clarify that probability tables must not contain non-zero probabilities for invalid values by @elasota in #3817
[x-compile] Fix cross-compiling for AArch64 with lld by @jcelerier in #3760
playTests.sh does no longer needs grep -E by @Cyan4973 in #3865
minor: playTests.sh more compatible with older versions of grep by @Cyan4973 in #3877
disable Intel CET Compatibility tests by @Cyan4973 in #3884
improve cmake test by @Cyan4973 in #3883
add sparc64 compilation test by @Cyan4973 in #3886
add a lorem ipsum generator by @Cyan4973 in #3890
Update Dependency in Intel CET Test; Re-Enable Test by @felixhandte in #3893
Improve compression of Arrays of Integers (High compression mode) by @Cyan4973 in #3895
[Zstd] Less verbose log for patch mode. by @sandreenko in #3899
fix 5921623844651008 by @Cyan4973 in #3900
Fix fuzz issue 5131069967892480 by @Cyan4973 in #3902
Advertise Availability of Security Vulnerability Notifications by @felixhandte in #3909
updated setup-msys2 to v2.22.0 by @Cyan4973 in #3914
Lorem Ipsum generator update by @Cyan4973 in #3913
Reduce scope of variables by @gruenich in #3903
Improve speed of ZSTD_c_targetCBlockSize by @Cyan4973 in #3915
More regular block sizes with targetCBlockSize by @Cyan4973 in #3917
removed sprintf usage from zstdcli.c by @Cyan4973 in #3916
Export a zstd::libzstd CMake target if only static or dynamic linkage is specified. by @teo-tsirpanis in #3811
fix version of actions/checkout by @Cyan4973 in #3926
minor Makefile refactoring by @Cyan4973 in #3753
lib/decompress: check for reserved bit corruption in zstd by @aimuz in #3840
Fix state table formatting by @elasota in #3816
Specify offset 0 as invalid and specify required fixup behavior by @elasota in #3824
update -V documentation by @Cyan4973 in #3928
fix LLU->ULL by @Cyan4973 in #3929
Fix building xxhash on AIX 5.1 by @likema in #3860
Fix building on HP-UX 11.11 PA-RISC by @likema in #3862
Fix AsyncIO reading seed queueing by @yoniko in #3940
Use ZSTD_LEGACY_SUPPORT=5 in "make test" by @embg in #3943
Pin sanitizer CI jobs to ubuntu-20.04 by @embg in #3945
chore: fix some typos by @acceptacross in #3949
new method to deal with offset==0 erroneous edge case by @Cyan4973 in #3937
add tests inspired from #2927 by @Cyan4973 in #3948
cmake refactor: move HP-UX specific logic into its own function by @Cyan4973 in #3946
Fix #3719 : mixing -c, -o and --rm by @Cyan4973 in #3942
minor: fix incorrect debug level by @Cyan4973 in #3936
add RISC-V emulation tests to Github CI by @Cyan4973 in #3934
prevent XXH64 from being autovectorized by XXH512 by default by @Cyan4973 in #3933
Stop Hardcoding the POSIX Version on BSDs by @felixhandte in #3952
Convert the CircleCI workflow to a GitHub Actions workflow by @jk0 in #3901
Add common compressed file types to --exclude-compressed by @daniellerozenblit in #3951
Export ZSTD_LEGACY_SUPPORT in tests/Makefile by @embg in #3955
Exercise ZSTD_findDecompressedSize() in the simple decompression fuzzer by @embg in #3959
Update ZSTD_RowFindBestMatch comment by @yoniko in #3947
Add the zeroSeq sample by @Cyan4973 in #3954
[cpu] Backport fix for rbx clobbering on Windows with Clang by @terrelln in #3957
Do not truncate file name in verbose mode by @Cyan4973 in #3956
updated documentation by @Cyan4973 in #3958
[asm][aarch64] Mark that BTI and PAC are supported by @terrelln in #3961
Use utimensat() on FreeBSD by @felixhandte in #3960
reduce the amount of #include in cover.h by @Cyan4973 in #3962
Remove Erroneous Exclusion of Hidden Files and Folders in --output-dir-mirror by @felixhandte in #3963
Promote ZSTD_c_targetCBlockSize Parameter to Stable API by @felixhandte in #3964
[cmake] Always create libzstd target by @terrelln in #3965
Remove incorrect docs regarding ZSTD_findFrameCompressedSize() by @embg in #3967
add line number to debug traces by @Cyan4973 in #3966
bump version number by @Cyan4973 in #3969
Export zstd's public headers via BUILD_INTERFACE by @terrelln in #3968
Fix bug with streaming decompression of magicless format by @embg in #3971
pzstd: use c++14 without conditions by @kanavin in #3682
Fix bugs in simple decompression fuzzer by @yoniko in #3978
Fuzzing and bugfixes for magicless-format decoding by @embg in #3976
Fix & fuzz ZSTD_generateSequences by @terrelln in #3981
Fail on errors when building fuzzers by @yoniko in #3979
[cmake] Emit warnings for contradictory build settings by @terrelln in #3975
Document the process for adding a new fuzzer by @embg in #3982
Fix -Werror=pointer-arith in fuzzers by @embg in #3983
Doc update by @Cyan4973 in #3977
v1.5.6 by @Cyan4973 in #3984
New Contributors
@Kim-SSi made their first contribution in #3600
@mredig made their first contribution in #3614
@dunhor made their first contribution in #3636
@sighingnow made their first contribution in #3657
@nidhijaju made their first contribution in #3688
@gjasny made their first contribution in #3665
@Coder-256 made their first contribution in #3700
@LocutusOfBorg made their first contribution in #3695
@void0red made their first contribution in #3704
@nikohoffren made their first contribution in #3701
@alexsifivetw made their first contribution in #3712
@0o001 made their first contribution in #3728
@QBos07 made their first contribution in #3720
@JohanMabille made their first contribution in #3739
@klausholstjacobsen made their first contribution in #3745
@Saverio976 made their first contribution in #3795
@elasota made their first contribution in #3806
@jondo2010 made their first contribution in #3812
@gruenich made their first contribution in #3807
@michoecho made their first contribution in #3847
@KapJI made their first contribution in #3850
@jcelerier made their first contribution in #3760
@sandreenko made their first contribution in #3899
@teo-tsirpanis made their first contribution in #3811
@aimuz made their first contribution in #3840
@acceptacross made their first contribution in #3949
@jk0 made their first contribution in #3901
Full Changelog: v1.5.5...v1.5.6Downloads
mirror of
https://github.com/facebook/zstd.git
synced 2025-03-04 16:15:54 +02:00