1
0
mirror of https://github.com/facebook/zstd.git synced 2025-03-07 01:10:04 +02:00

353 Commits

Author SHA1 Message Date
Johan Mabille
253873220f Fixed zstd cmake shared build on windows 2023-08-27 16:51:24 +02:00
Mustafa UZUN
a02d81f944 fix: ZSTD_BUILD_DECOMPRESSION message 2023-08-19 15:44:53 +03:00
Gregor Jasny
7e09f07b32 Fix Intel Xcode builds with assembly
When forcing the source file language to `C`, Xcode enforces
the file to be compiled as `C` by appending `-x c` to the
compiler command line.

For now try to limit the damage and only enforce the language
if the ASM and C compilers differ.

Reproducer (CMake `3.26.4`, Xcode `14.3`):
```
cmake -S build/cmake -B _b -GXcode -DCMAKE_OSX_ARCHITECTURES=x86_64
cmake --build _b
```

Fix: #3622
2023-06-02 12:43:59 +02:00
Tao He
5059618295 Add options in Makefile to cmake
Signed-off-by: Tao He <sighingnow@gmail.com>
2023-05-26 13:45:21 -07:00
Tobias Hieta
979b047114 Disable linker flag detection on MSVC/ClangCL.
This fixes compilation with clang-cl on Windows. There
is a bug in cmake so that check_linker_flag() doesn't give
the correct result when using link.exe/lld-link.exe.

Details in CMake's gitlab: https://gitlab.kitware.com/cmake/cmake/-/issues/22023

Fixes #3522
2023-03-22 22:13:57 +01:00
Yann Collet
4ebaf36582
Merge pull request #3490 from eli-schwartz/meson-tests-noprograms
meson: always build the zstd binary when tests are enabled
2023-02-16 11:27:27 -08:00
Sutou Kouhei
8420502ef9 Don't require CMake 3.18 or later
fix #3500

CMake 3.18 or later was required by #3392. Because it uses
`CheckLinkerFlag`. But requiring CMake 3.18 or later is a bit
aggressive. Because Ubuntu 20.04 LTS still uses CMake 3.16.3:
https://packages.ubuntu.com/search?keywords=cmake

This change disables `-z noexecstack` check with old CMake. This will
not break any existing users. Because users who need `-z noexecstack`
must already use CMake 3.18 or later.
2023-02-16 10:08:45 -08:00
Elliot Gorokhovsky
a7de1d9f49
Fix all MSVC warnings (#3495)
* fix and test MSVC AVX2 build

* treat msbuild warnings as errors

* fix incorrect MSVC 2019 compiler warning

* fix MSVC error D9035: option 'Gm' has been deprecated and will be removed in a future release
2023-02-11 10:56:59 -05:00
Eli Schwartz
97ab0e2ab6
meson: always build the zstd binary when tests are enabled
We need to run it for the tests, even if programs are disabled. So if
they are disabled, create a build rule for the program, but don't
install it. Just make it available for the test itself.
2023-02-10 00:33:07 -05:00
Eli Schwartz
183a18a45c
meson: correctly specify the dependency relationship for playtests
It depends on the zstd program being built, and passes it as an env
variable. Just like datagen. But for datagen, we explicitly depend on
it, while for zstd, we assume it's built as part of "all".

This can be wrong in two cases:
- when running individual tests, meson can (re)build just what is needed
  for that one test
- a later patch will handle building zstd but not by default
2023-02-10 00:33:07 -05:00
Elliot Gorokhovsky
2a402626dd
External matchfinder API (#3333)
* First building commit with sample matchfinder

* Set up ZSTD_externalMatchCtx struct

* move seqBuffer to ZSTD_Sequence*

* support non-contiguous dictionary

* clean up parens

* add clearExternalMatchfinder, handle allocation errors

* Add useExternalMatchfinder cParam

* validate useExternalMatchfinder cParam

* Disable LDM + external matchfinder

* Check for static CCtx

* Validate mState and mStateDestructor

* Improve LDM check to cover both branches

* Error API with optional fallback

* handle RLE properly for external matchfinder

* nit

* Move to a CDict-like model for resource ownership

* Add hidden useExternalMatchfinder bool to CCtx_params_s

* Eliminate malloc, move to cwksp allocation

* Handle CCtx reset properly

* Ensure seqStore has enough space for external sequences

* fix capitalization

* Add DEBUGLOG statements

* Add compressionLevel param to matchfinder API

* fix c99 issues and add a param combination error code

* nits

* Test external matchfinder API

* C90 compat for simpleExternalMatchFinder

* Fix some @nocommits and an ASAN bug

* nit

* nit

* nits

* forward declare copySequencesToSeqStore functions in zstd_compress_internal.h

* nit

* nit

* nits

* Update copyright headers

* Fix CMake zstreamtest build

* Fix copyright headers (again)

* typo

* Add externalMatchfinder demo program to make contrib

* Reduce memory consumption for small blockSize

* ZSTD_postProcessExternalMatchFinderResult nits

* test sum(matchlen) + sum(litlen) == srcSize in debug builds

* refExternalMatchFinder -> registerExternalMatchFinder

* C90 nit

* zstreamtest nits

* contrib nits

* contrib nits

* allow block splitter + external matchfinder, refactor

* add windowSize param

* add contrib/externalMatchfinder/README.md

* docs

* go back to old RLE heuristic because of the first block issue

* fix initializer element is not a constant expression

* ref contrib from zstd.h

* extremely pedantic compiler warning fix, meson fix, typo fix

* Additional docs on API limitations

* minor nits

* Refactor maxNbSeq calculation into a helper function

* Fix copyright
2022-12-28 16:45:14 -05:00
Yann Collet
6640377783 cmake build: fix nit
reported by @jaimeMF in https://github.com/facebook/zstd/pull/3392#discussion_r1056643794
2022-12-23 14:18:11 -08:00
Nick Terrell
31a703ec13 [cmake] Add noexecstack to compiler/linker flags 2022-12-21 17:30:14 -08:00
Nick Terrell
651a381060 [build][cmake] Fix cmake with custom assembler
Tell CMake to explicitly compile our assembly as C code, because we
require it is compiled by a C compiler, and it is only enabled for
clang/gcc.

Fixes #3193.
2022-12-20 13:02:21 -08:00
W. Felix Handte
5d693cc38c Coalesce Almost All Copyright Notices to Standard Phrasing
```
for f in $(find . \( -path ./.git -o -path ./tests/fuzz/corpora -o -path ./tests/regression/data-cache -o -path ./tests/regression/cache \) -prune -o -type f); do sed -i '/Copyright .* \(Yann Collet\)\|\(Meta Platforms\)/ s/Copyright .*/Copyright (c) Meta Platforms, Inc. and affiliates./' $f; done

git checkout HEAD -- build/VS2010/libzstd-dll/libzstd-dll.rc build/VS2010/zstd/zstd.rc tests/test-license.py contrib/linux-kernel/test/include/linux/xxhash.h examples/streaming_compression_thread_pool.c lib/legacy/zstd_v0*.c lib/legacy/zstd_v0*.h
nano ./programs/windres/zstd.rc
nano ./build/VS2010/zstd/zstd.rc
nano ./build/VS2010/libzstd-dll/libzstd-dll.rc
```
2022-12-20 12:52:34 -05:00
W. Felix Handte
7f12f24cf4 Rewrite Copyright Date Ranges from -present to -2022
Apparently it's better. Somehow.

```
for f in $(find . \( -path ./.git -o -path ./tests/fuzz/corpora -o -path ./tests/regression/data-cache -o -path ./tests/regression/cache \) -prune -o -type f); do echo $f; sed -i 's/\-present/-2022/' $f; done

g co HEAD -- build/meson/
```
2022-12-20 12:44:56 -05:00
W. Felix Handte
8927f985ff Update Copyright Headers 'Facebook' -> 'Meta Platforms'
```
for f in $(find . \( -path ./.git -o -path ./tests/fuzz/corpora \) -prune -o -type f);
do
  sed -i 's/Facebook, Inc\./Meta Platforms, Inc. and affiliates./' $f;
done
```
2022-12-20 12:37:57 -05:00
W. Felix Handte
a05cca6208 Manually Update VS Code Copyright Definitions 2022-12-20 12:36:06 -05:00
Yann Collet
6a90c0f6fb
Merge pull request #3289 from nmoinvaz/cmake/playtest-win
Don't attempt playTests.sh cmake test if running on Windows.
2022-12-19 16:39:38 -08:00
Yonatan Komornik
aaa38b29bb meson: zstreamtests should now pass on Windows 2022-12-17 13:38:02 -08:00
Eli Schwartz
6747ba4ef5
meson: mark a known test failure on Windows 2022-12-16 14:35:31 -05:00
Eli Schwartz
9c3e18f7fe
meson: add support for running both fast and slow version of tests
playTests.sh has an option to run really slow tests. This is enabled by
default in Meson, but what we really want is to do like the Makefile,
and run the fast ones by default, but with an option to run the slow
ones instead.
2022-12-16 14:27:46 -05:00
Eli Schwartz
26134b4565
meson: don't require valgrind tests
It's entirely possible some people don't have valgrind installed, but
still want to run the tests. If they don't have it installed, then they
probably don't intend to run those precise test targets anyway.

Also, this solves an error when running the tests in an automated
environment. The valgrind tests have a hard dependency on behavior such
as `./zstd` erroring out with the message "stdin is a console, aborting"
which does not work if the automated environment doesn't have a console.
As a rough heuristic, automated environments lacking a console will
*probably* also not have valgrind, so avoiding that test definition
neatly sidesteps the issue.

Also, valgrind is not easily installable on macOS, at least homebrew
says it isn't available there. This makes it needlessly hard to
enable the testsuite on macOS.
2022-12-16 14:27:46 -05:00
Eli Schwartz
626425dce0
meson: fix warning for using too-new features
In commit 031de3c69ccbf3282ed02fb49369b476730aeca8 a feature of Meson
0.50.0 was added, but the minimum specified version of Meson is 0.48.0.
Meson therefore emitted a warning:

WARNING: Project targets '>=0.48.0' but uses feature introduced in '0.50.0': required arg in compiler.has_header.

And if anyone actually used Meson 0.48.0 to build with, it would error
out with mysterious claims that the build file itself is invalid, rather
than telling the user to install a newer version of Meson.

Solve this by bumping the minimum version to align with reality. This
e.g. drops support for Debian oldstable (buster)'s packaged version of
Meson, but still works if backports are enabled, or if the user can
`pip install` a newer version.
2022-12-15 20:08:35 -05:00
Eli Schwartz
e8401e9e8d
meson: fix broken commit that broke the build
In commit 031de3c69ccbf3282ed02fb49369b476730aeca8 some code was added
that returned a boolean, but was treated as if it returned a dependency
object. This wasn't tested and could not work. Moreover, zstd no longer
built at all unless the entire programs directory was disabled and not
even evaluated.

Fix the return type checking.
2022-12-15 20:08:28 -05:00
Eli Schwartz
e0ef09ddba meson: partial fix for building pzstd on MSVC
It uses non-portable compiler options unconditionally. Elsewhere, we
check the compiler ID and only add the right ones, globally. Do the same
here.

NDEBUG can actually be handled by a core option, so while we are moving
things around, do so.

Unfortunately, this doesn't fix things entirely. The remaining issue is
not Meson's issue though -- MSVC simply does not like this source code
and somehow chokes on innocent code with the inscrutable "syntax error"
and "illegal token".
2022-12-15 15:56:06 -08:00
daniellerozenblit
e2fc93340f
Merge branch 'dev' into http-to-https 2022-12-15 10:46:13 -05:00
Rosen Penev
031de3c69c meson: make backtrace dependency on execinfo
musl libc for example has no such header.

Signed-off-by: Rosen Penev <rosenp@gmail.com>
2022-12-14 15:39:51 -08:00
Danielle Rozenblit
4dffc35f2e Convert references to https from http 2022-12-14 06:58:35 -08:00
Eli Schwartz
6c3ed93c27
meson: get version up front
Run the scraper command to establish the project version immediately,
rather than wait for the build to be configured. This simplifies the
code and ensures that project introspection works correctly.
2022-11-30 21:36:24 -05:00
Nathan Moinvaziri
b87f3102ff Only run playTests.sh cmake test if in unix shell environment. 2022-10-15 09:21:14 -07:00
Andrea Pappacoda
a0b09d0ff7
build(cmake): improve pkg-config generation
With this patch the pkg-config generation when using the CMake build
system is improved in the following ways:

- Libs.private is now filled when needed
- The JoinPaths module is now used to join paths, leading to simpler
  code
- The .pc file is always generated, regardless of the platform, as it
  can also be consumed on Windows

Here's how the .pc file is affected by these changes, in comparison to
the one generated with the official Makefiles:

    $ diff -s lib/libzstd.pc build/cmake/build-old/lib/libzstd.pc
    15c15
    < Libs.private: -pthread
    ---
    > Libs.private:

    $ diff -s lib/libzstd.pc build/cmake/build-new/lib/libzstd.pc
    Files lib/libzstd.pc and build/cmake/build-new/lib/libzstd.pc are
    identical
2022-08-28 13:24:30 +02:00
William Ahern
60fcc36af6 escape glob pattern special characters in subject string before generating search patterns in combine.sh list_has_item 2022-08-10 20:58:55 -07:00
William Ahern
cca3544708 restore combine.sh bash performance while still sticking to POSIX 2022-08-10 16:51:17 -07:00
Taylor Braun-Jones
cd9d0a7e6e Fix ZSTD_BUILD_TESTS=ON build with MSVC
Fixes:

    Command line error D8021 : invalid numeric argument '/Wno-deprecated-declarations'
2022-06-30 13:20:42 -04:00
Yann Collet
15f3605135 removed gnu99 statement from meson recipe 2022-06-20 18:18:40 -07:00
Yann Collet
eceecc5b2c removed explicit compilation standard from cmake script
it's not expected to be useful
and can actually lead to subtle side effects
such as #3163.
2022-06-19 14:52:32 -07:00
Eli Schwartz
6548ec7440
meson: for internal linkage, link to both libzstd and a static copy of it
Partial, Meson-only implementation of #2976 for non-MSVC builds.

Due to the prevalence of private symbol reuse, linking to a shared
library is simply utterly unreliable, but we still want to defer to the
shared library for installable applications. By linking to both, we can
share symbols where possible, and statically link where needed.

This means we no longer need to manually track every file that needs to
be extracted and reused.

The flip side is that MSVC completely does not support this, so for MSVC
builds we just link to a full static copy even where
-Ddefault_library=shared.

As a side benefit, by using library inclusion rather than including
extra explicit object files, the zstd program shrinks in size slightly
(~4kb).
2022-04-28 21:57:02 -04:00
Eli Schwartz
8d522b8a9d
meson: avoid rebuilding some libzstd sources in the programs
These need to be explicitly included as we use their private symbols,
but we don't need to recompile them when we can reuse the existing
objects.

Minus 7 compile steps.
2022-04-28 21:56:36 -04:00
Eli Schwartz
df6eefb3bb
meson: avoid rebuilding some libzstd files in the test programs
The poolTests program already linked to libzstd, and later to
libtestcommon with included libzstd objects. So this was redundant.

Minus 4 compile steps.
2022-04-28 21:56:36 -04:00
Yann Collet
66633f9386
Merge pull request #3039 from eli-schwartz/meson
Meson fixups for Windows
2022-04-19 15:51:19 -07:00
Dominique Pelle
b772f53952 Typo and grammar fixes 2022-03-12 08:58:04 +01:00
Yann Collet
db104f6e83
Merge pull request #3079 from niamster/cmake-compat-5
build:cmake: enable ZSTD legacy support by default
2022-03-08 00:21:23 -08:00
Dimitris Apostolou
cf1894b324
Fix typos 2022-03-05 23:47:25 +02:00
Dmytro Milinevskyi
03bba1b0bf build:cmake: enable ZSTD legacy support by default 2022-03-01 18:29:47 +01:00
Alexander Shadchin
317bd108fe
Select legacy level for cmake 2022-02-04 14:24:58 +03:00
Eli Schwartz
ef78b9af30
meson: valgrind wrapper should return correct errors
While trying to raise an exception on failures, it instead raised an
exception for misusing the exception. CalledProcessError is only
supposed to be used when given a return code and a command, and it
prints:

Command '{cmd}' returned non-zero exit status {ret}

Passing an error message string instead, just errored out with:

TypeError: __init__() missing 1 required positional argument

Instead use the subprocess module's base error which does accept string
messages. Everything that used to error out, still errors out, but now
they do so with a slightly prettier console message.
2022-01-30 21:50:42 -05:00
Eli Schwartz
84c05453db
meson: never require a libm
libm is not guaranteed to exist. POSIX requires the math functions to
exist, but doesn't require to have it be a standalone library.

On platforms where libm exists as a standalone library, it will always
be found by meson -- it is shipped with libc.

If it is not found, then we can safely assume the linker will make the
math functions available by default.

See https://mesonbuild.com/howtox.html#add-math-library-lm-portably

Fixes building with bin_tests=true on Windows.
2022-01-30 21:50:42 -05:00
Eli Schwartz
5b2c6c776a
meson: fix resource file compilation on Windows
It needs to know about the correct include directories on its own.
2022-01-30 21:50:42 -05:00
Yonatan Komornik
70df5de1b2
AsyncIO compression part 1 - refactor of existing asyncio code (#3021)
* Refactored fileio.c:
- Extracted asyncio code to fileio_asyncio.c/.h
- Moved type definitions to fileio_types.h
- Moved common macro definitions needed by both fileio.c and fileio_asyncio.c to fileio_common.h

* Bugfix - rename fileio_asycio to fileio_asyncio

* Added copyrights & license to new files

* CR fixes
2022-01-24 14:43:02 -08:00