krak/zstd
krak/zstd
1
0
Fork 0
mirror of https://github.com/facebook/zstd.git synced 2025-03-07 09:26:03 +02:00
Code Issues Releases 1 Activity
10,860 Commits 25 Branches 76 Tags
Commit Graph

10860 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Yann Collet
a8b86d024a refactor documentation of the FSE decoding table build process 2024-10-02 23:09:06 -07:00
Yann Collet
75b0f5f4f5
Merge pull request #4153 from artem/fix-meson-includes 
meson: Do not export private headers in libzstd_dep to avoid name clash
 2024-10-02 16:51:44 -07:00
Yann Collet
dda3cdfdec
Merge pull request #4156 from facebook/rm_circleci 
removing nightly tests built on circleci
 2024-10-02 16:51:15 -07:00
Yann Collet
751bf1ffd8
Merge pull request #4157 from facebook/fix_result_c 
fix incorrect pointer manipulation
 2024-10-02 16:50:45 -07:00
Yann Collet
dcc8fd0472
Merge pull request #4158 from facebook/benchzstd_fclose 
fix missing fclose()
 2024-10-02 16:49:43 -07:00
Yann Collet
8edd147686 fix missing fclose() 
fix #4151
 2024-10-01 09:52:45 -07:00
Yann Collet
de6cc98e07 fix incorrect pointer manipulation 
fix #4155
 2024-10-01 09:25:26 -07:00
Yann Collet
3d5d3f5630 removing nightly tests built on circleci 2024-09-30 21:38:29 -07:00
Yann Collet
27bf1362fe
Merge pull request #4154 from dearblue/freebsd-14.1 
Update FreeBSD VM image to 14.1
 2024-09-30 11:54:32 -07:00
Artem Labazov
ccc02a9a77 meson: Fix contrib and tests build 2024-09-30 18:05:57 +03:00
Artem Labazov
d2d49a1161 meson: Do not export private headers in libzstd_dep to avoid name clash 
This way libzstd_dep does not override, for instance, <xxhash.h>
 2024-09-30 17:03:42 +03:00
dearblue
a3b5c4521c Update FreeBSD VM image to 14.1 
FreeBSD 14.0 will reach the end of life on 2024-09-30.
The updated 14.1 is scheduled to end-of-life on 2025-03-31.

ref. https://www.freebsd.org/releases/14.2R/schedule/
 2024-09-30 22:45:17 +09:00
Yann Collet
984d11a4d1
Merge pull request #4146 from facebook/dictBench_Doc 
update documentation: specify that Dictionary can be used for benchmark
 2024-09-27 13:44:42 -07:00
Yann Collet
d2212c680a
Merge pull request #4013 from elasota/spec-clarify-offset-code-overflow 
Specify that decoders may reject non-zero probabilities for larger offset codes than implementation supports
 2024-09-27 13:42:32 -07:00
Yann Collet
039f404faa update documentation to specify that Dictionary can be used for benchmark 
fix #4139
 2024-09-25 16:56:01 -07:00
inventor500
9215de52c7 Included suggestion from @neheb 2024-09-25 09:51:05 -07:00
inventor500
a8b544d460 Fixed warning when compiling pzstd with CPPFLAGS=-Wunused-result and CXXFLAGS=-std=c++17 2024-09-25 09:51:05 -07:00
Yann Collet
bc96d4b077
Merge pull request #4119 from xionghul/dev 
Fix zstd-pgo run error
 2024-09-24 17:55:43 -07:00
Yann Collet
d27a4cd4ac
Merge pull request #4143 from facebook/fix_dictsizemin_dic 
fix doc nit: ZDICT_DICTSIZE_MIN
 2024-09-24 17:55:25 -07:00
Ilya Tokar
e8fce38954 Optimize compression by avoiding unpredictable branches 
Avoid unpredictable branch. Use conditional move to generate the address
that is guaranteed to be safe and compare unconditionally.
Instead of

if (idx < limit && x[idx] == val ) // mispredicted idx < limit branch

Do

addr = cmov(safe,x+idx)
if (*addr == val && idx < limit) // almost always false so well predicted

Using microbenchmarks from https://github.com/google/fleetbench,
I get about ~10% speed-up:

name                                                                                          old cpu/op   new cpu/op    delta
BM_ZSTD_COMPRESS_Fleet/compression_level:-7/window_log:15                                     1.46ns ± 3%   1.31ns ± 7%   -9.88%  (p=0.000 n=35+38)
BM_ZSTD_COMPRESS_Fleet/compression_level:-7/window_log:16                                     1.41ns ± 3%   1.28ns ± 3%   -9.56%  (p=0.000 n=36+39)
BM_ZSTD_COMPRESS_Fleet/compression_level:-5/window_log:15                                     1.61ns ± 1%   1.43ns ± 3%  -10.70%  (p=0.000 n=30+39)
BM_ZSTD_COMPRESS_Fleet/compression_level:-5/window_log:16                                     1.54ns ± 2%   1.39ns ± 3%   -9.21%  (p=0.000 n=37+39)
BM_ZSTD_COMPRESS_Fleet/compression_level:-3/window_log:15                                     1.82ns ± 2%   1.61ns ± 3%  -11.31%  (p=0.000 n=37+40)
BM_ZSTD_COMPRESS_Fleet/compression_level:-3/window_log:16                                     1.73ns ± 3%   1.56ns ± 3%   -9.50%  (p=0.000 n=38+39)
BM_ZSTD_COMPRESS_Fleet/compression_level:-1/window_log:15                                     2.12ns ± 2%   1.79ns ± 3%  -15.55%  (p=0.000 n=34+39)
BM_ZSTD_COMPRESS_Fleet/compression_level:-1/window_log:16                                     1.99ns ± 3%   1.72ns ± 3%  -13.70%  (p=0.000 n=38+38)
BM_ZSTD_COMPRESS_Fleet/compression_level:0/window_log:15                                      3.22ns ± 3%   2.94ns ± 3%   -8.67%  (p=0.000 n=38+40)
BM_ZSTD_COMPRESS_Fleet/compression_level:0/window_log:16                                      3.19ns ± 4%   2.86ns ± 4%  -10.55%  (p=0.000 n=40+38)
BM_ZSTD_COMPRESS_Fleet/compression_level:1/window_log:15                                      2.60ns ± 3%   2.22ns ± 3%  -14.53%  (p=0.000 n=40+39)
BM_ZSTD_COMPRESS_Fleet/compression_level:1/window_log:16                                      2.46ns ± 3%   2.13ns ± 2%  -13.67%  (p=0.000 n=39+36)
BM_ZSTD_COMPRESS_Fleet/compression_level:2/window_log:15                                      2.69ns ± 3%   2.46ns ± 3%   -8.63%  (p=0.000 n=37+39)
BM_ZSTD_COMPRESS_Fleet/compression_level:2/window_log:16                                      2.63ns ± 3%   2.36ns ± 3%  -10.47%  (p=0.000 n=40+40)
BM_ZSTD_COMPRESS_Fleet/compression_level:3/window_log:15                                      3.20ns ± 2%   2.95ns ± 3%   -7.94%  (p=0.000 n=35+40)
BM_ZSTD_COMPRESS_Fleet/compression_level:3/window_log:16                                      3.20ns ± 4%   2.87ns ± 4%  -10.33%  (p=0.000 n=40+40)

I've also measured the impact on internal workloads and saw similar
~10% improvement in performance, measured by cpu usage/byte of data.
 2024-09-20 16:07:01 -04:00
Yann Collet
7a48dc230c fix doc nit: ZDICT_DICTSIZE_MIN 
fix #4142
 2024-09-19 09:50:30 -07:00
Yann Collet
20707e3718
Merge pull request #4129 from facebook/mitigate_32bit 
Limit range of operations on Indexes in 32-bit mode
 2024-08-22 11:00:50 -07:00
Yann Collet
09cb37cbb1 Limit range of operations on Indexes in 32-bit mode 
and use unsigned type.
This reduce risks that an operation produces a negative number when crossing the 2 GB limit.
 2024-08-21 11:03:43 -07:00
Yann Collet
ad038d8768
Merge pull request #4128 from facebook/dependabot/github_actions/github/codeql-action-3.26.2 
Bump github/codeql-action from 3.25.1 to 3.26.2
 2024-08-19 13:54:06 -07:00
dependabot[bot]
ec0c41414d
Bump github/codeql-action from 3.25.1 to 3.26.2 
Bumps [github/codeql-action](https://github.com/github/codeql-action) from 3.25.1 to 3.26.2.
- [Release notes](https://github.com/github/codeql-action/releases)
- [Changelog](https://github.com/github/codeql-action/blob/main/CHANGELOG.md)
- [Commits](c7f9125735...429e197704)

---
updated-dependencies:
- dependency-name: github/codeql-action
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
 2024-08-19 05:37:33 +00:00
Yann Collet
a761013b03
Merge pull request #4122 from facebook/dependabot/github_actions/actions/setup-java-4 
Bump actions/setup-java from 3 to 4
 2024-08-11 23:07:43 -07:00
dependabot[bot]
aed3c7540a
Bump actions/setup-java from 3 to 4 
Bumps [actions/setup-java](https://github.com/actions/setup-java) from 3 to 4.
- [Release notes](https://github.com/actions/setup-java/releases)
- [Commits](https://github.com/actions/setup-java/compare/v3...v4)

---
updated-dependencies:
- dependency-name: actions/setup-java
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
 2024-08-12 05:16:05 +00:00
Yann Collet
1eb32ff594
Merge pull request #4115 from Adenilson/leak01 
[zstd][leak] Avoid memory leak on early return of ZSTD_generateSequence
 2024-08-09 14:09:17 -07:00
Yann Collet
ee1fc7ee5c
Merge pull request #4114 from Adenilson/trace01 
[riscv] Enable support for weak symbols
 2024-08-09 14:08:57 -07:00
Yann Collet
10e2a8042c
Merge pull request #4113 from facebook/fix4110 
formatString_u() can display numbers > 100
 2024-08-08 17:19:08 -07:00
Xionghu Luo
bf4a43fcd4 Fix zstd-pgo run error 
The -Werror=missing-profile caused thread/zlib/lzma/lz4 detection failure
when build with profile-use, thus caused ZSTD_MULTITHREAD etc. is not
defined for profile-use, then there will be many profile mismatch information
in output and the final binary reports run error sometimes as below:

Error : ZSTD_CCtx_setParameter(ctx, ZSTD_c_nbWorkers, adv->nbWorkers) failed : Unsupported parameter

Signed-off-by: Xionghu Luo <xionghuluo@tencent.com>
 2024-08-08 15:13:27 +08:00
Yann Collet
9a5c74b2ca
Merge pull request #4109 from facebook/actionsTest 
added android-ndk-build
 2024-08-07 18:17:18 -07:00
Adenilson Cavalcanti
a40bad8ec0 [zstd][leak] Avoid memory leak on early return of ZSTD_generateSequence 
Sanity checks on a few of the context parameters (i.e. workers and block size)
may prompt an early return on ZSTD_generateSequences.

Allocating the destination buffer past those return points avoids a potential
memory leak.

This patch should fix issue #4112.
 2024-08-06 18:01:20 -07:00
Adenilson Cavalcanti
6dbd49bcd0 [riscv] Enable support for weak symbols 
Both gcc and clang support weak symbols on RISC-V, therefore
let's enable it.

This should fix issue #4069.
 2024-08-06 16:55:32 -07:00
Yann Collet
14b8d398fd fix c90 comment style 2024-08-06 12:47:30 -07:00
Yann Collet
89451cafbd formatString_u() can display numbers > 100 
fixes #4110
 2024-08-06 11:44:37 -07:00
Yann Collet
6b16169ccf
Merge pull request #4111 from facebook/dependabot/github_actions/msys2/setup-msys2-2.24.1 
Bump msys2/setup-msys2 from 2.24.0 to 2.24.1
 2024-08-05 00:39:06 -07:00
dependabot[bot]
688a815c86
Bump msys2/setup-msys2 from 2.24.0 to 2.24.1 
Bumps [msys2/setup-msys2](https://github.com/msys2/setup-msys2) from 2.24.0 to 2.24.1.
- [Release notes](https://github.com/msys2/setup-msys2/releases)
- [Changelog](https://github.com/msys2/setup-msys2/blob/main/CHANGELOG.md)
- [Commits](5df0ca6cbf...ddf331adae)

---
updated-dependencies:
- dependency-name: msys2/setup-msys2
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
 2024-08-05 05:17:43 +00:00
Yann Collet
fdfb2aff39
Merge pull request #4102 from proppy/decodecorpus-ext 
tests/decodecorpus: add more advanced options
 2024-07-31 01:43:15 -07:00
Yann Collet
cb784edf5d added android-ndk-build 2024-07-30 11:34:49 -07:00
Yann Collet
d4b176db0f
Merge pull request #4107 from Adenilson/ndk01 
[zstd][android] Fix build with NDK r27
 2024-07-30 11:34:27 -07:00
Adenilson Cavalcanti
c3c28c4d5a [zstd][android] Fix build with NDK r27 
The NDK cross compiler declares the target as __linux (which is
not technically incorrect), which triggers the enablement of _GNU_SOURCE
in the newly added code that requires the presence of qsort_r() used
in the COVER dictionary code.

Even though the NDK uses llvm/libc, it doesn't declare qsort_r()
in the stdlib.h header.

The build fix is to only activate the _GNU_SOURCE macro if the OS is
*not* Android, as then we will fallback to the C90 compliant code.

This patch should solve the reported issue number #4103.
 2024-07-29 17:13:58 -07:00
Yann Collet
43b713a2ce
Merge pull request #4094 from RubenKelevra/patchfrom_singlethread_man_update 
clarify when to use '--single-thread' with '--patch-from'
 2024-07-29 09:34:55 -07:00
Yann Collet
81b52f31e3
Merge pull request #4106 from facebook/dependabot/github_actions/msys2/setup-msys2-2.24.0 
Bump msys2/setup-msys2 from 2.23.0 to 2.24.0
 2024-07-29 09:34:17 -07:00
Yann Collet
adcb543fb8
Merge pull request #4087 from jclab-joseph/fix/genhtml-windows 
gen_html: Fix build error with mingw
 2024-07-29 09:18:27 -07:00
Yann Collet
b768e09393
Merge pull request #4101 from uilianries/readme/conan 
[docs] Add instruction how to install zstd using Conan
 2024-07-29 09:17:51 -07:00
Yann Collet
80b1385d23
Merge pull request #4104 from facebook/dependabot/github_actions/ossf/scorecard-action-2.4.0 
Bump ossf/scorecard-action from 2.3.1 to 2.4.0
 2024-07-29 09:17:11 -07:00
dependabot[bot]
46a3135524
Bump msys2/setup-msys2 from 2.23.0 to 2.24.0 
Bumps [msys2/setup-msys2](https://github.com/msys2/setup-msys2) from 2.23.0 to 2.24.0.
- [Release notes](https://github.com/msys2/setup-msys2/releases)
- [Changelog](https://github.com/msys2/setup-msys2/blob/main/CHANGELOG.md)
- [Commits](d0e80f58df...5df0ca6cbf)

---
updated-dependencies:
- dependency-name: msys2/setup-msys2
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
 2024-07-29 05:34:42 +00:00
dependabot[bot]
efbb5ef015
Bump ossf/scorecard-action from 2.3.1 to 2.4.0 
Bumps [ossf/scorecard-action](https://github.com/ossf/scorecard-action) from 2.3.1 to 2.4.0.
- [Release notes](https://github.com/ossf/scorecard-action/releases)
- [Changelog](https://github.com/ossf/scorecard-action/blob/main/RELEASE.md)
- [Commits](0864cf1902...62b2cac7ed)

---
updated-dependencies:
- dependency-name: ossf/scorecard-action
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
 2024-07-29 05:34:32 +00:00
Pawel Czarnecki
1f5df587fa tests/decodecorpus: add more advanced options 
- add option to force a specific block type
- add option to force a specific literal tyle
- add option to generate only frame headers
- add option to skip generating magic numbers

Co-authored-by: Maciej Dudek <mdudek@antmicro.com>
Co-authored-by: Pawel Czarnecki <pczarnecki@antmicro.com>
Co-authored-by: Robert Winkler <rwinkler@antmicro.com>
Co-authored-by: Roman Dobrodii <rdobrodii@antmicro.com>
 2024-07-26 09:47:16 +02:00