krak/zstd
krak/zstd
1
0
Fork 0
mirror of https://github.com/facebook/zstd.git synced 2025-03-07 09:26:03 +02:00
Code Issues Releases 1 Activity
10,054 Commits 25 Branches 76 Tags
Commit Graph

222 Commits

Author SHA1 Message Date
Elliot Gorokhovsky
ff42ed1582
Rename "External Matchfinder" to "Block-Level Sequence Producer" (#3484) 
* change "external matchfinder" to "external sequence producer"

* migrate contrib/ to new naming convention

* fix contrib build

* fix error message

* update debug strings

* fix def of invalid sequences in zstd.h

* nit

* update CHANGELOG

* fix .gitignore
 2023-02-09 17:01:17 -05:00
Yann Collet
ea684c335a added c89 build test to CI 2023-01-19 14:59:30 -08:00
Elliot Gorokhovsky
2a402626dd
External matchfinder API (#3333) 
* First building commit with sample matchfinder

* Set up ZSTD_externalMatchCtx struct

* move seqBuffer to ZSTD_Sequence*

* support non-contiguous dictionary

* clean up parens

* add clearExternalMatchfinder, handle allocation errors

* Add useExternalMatchfinder cParam

* validate useExternalMatchfinder cParam

* Disable LDM + external matchfinder

* Check for static CCtx

* Validate mState and mStateDestructor

* Improve LDM check to cover both branches

* Error API with optional fallback

* handle RLE properly for external matchfinder

* nit

* Move to a CDict-like model for resource ownership

* Add hidden useExternalMatchfinder bool to CCtx_params_s

* Eliminate malloc, move to cwksp allocation

* Handle CCtx reset properly

* Ensure seqStore has enough space for external sequences

* fix capitalization

* Add DEBUGLOG statements

* Add compressionLevel param to matchfinder API

* fix c99 issues and add a param combination error code

* nits

* Test external matchfinder API

* C90 compat for simpleExternalMatchFinder

* Fix some @nocommits and an ASAN bug

* nit

* nit

* nits

* forward declare copySequencesToSeqStore functions in zstd_compress_internal.h

* nit

* nit

* nits

* Update copyright headers

* Fix CMake zstreamtest build

* Fix copyright headers (again)

* typo

* Add externalMatchfinder demo program to make contrib

* Reduce memory consumption for small blockSize

* ZSTD_postProcessExternalMatchFinderResult nits

* test sum(matchlen) + sum(litlen) == srcSize in debug builds

* refExternalMatchFinder -> registerExternalMatchFinder

* C90 nit

* zstreamtest nits

* contrib nits

* contrib nits

* allow block splitter + external matchfinder, refactor

* add windowSize param

* add contrib/externalMatchfinder/README.md

* docs

* go back to old RLE heuristic because of the first block issue

* fix initializer element is not a constant expression

* ref contrib from zstd.h

* extremely pedantic compiler warning fix, meson fix, typo fix

* Additional docs on API limitations

* minor nits

* Refactor maxNbSeq calculation into a helper function

* Fix copyright
 2022-12-28 16:45:14 -05:00
W. Felix Handte
5d693cc38c Coalesce Almost All Copyright Notices to Standard Phrasing 
```
for f in $(find . \( -path ./.git -o -path ./tests/fuzz/corpora -o -path ./tests/regression/data-cache -o -path ./tests/regression/cache \) -prune -o -type f); do sed -i '/Copyright .* \(Yann Collet\)\|\(Meta Platforms\)/ s/Copyright .*/Copyright (c) Meta Platforms, Inc. and affiliates./' $f; done

git checkout HEAD -- build/VS2010/libzstd-dll/libzstd-dll.rc build/VS2010/zstd/zstd.rc tests/test-license.py contrib/linux-kernel/test/include/linux/xxhash.h examples/streaming_compression_thread_pool.c lib/legacy/zstd_v0*.c lib/legacy/zstd_v0*.h
nano ./programs/windres/zstd.rc
nano ./build/VS2010/zstd/zstd.rc
nano ./build/VS2010/libzstd-dll/libzstd-dll.rc
```
 2022-12-20 12:52:34 -05:00
W. Felix Handte
7f12f24cf4 Rewrite Copyright Date Ranges from -present to -2022 
Apparently it's better. Somehow.

```
for f in $(find . \( -path ./.git -o -path ./tests/fuzz/corpora -o -path ./tests/regression/data-cache -o -path ./tests/regression/cache \) -prune -o -type f); do echo $f; sed -i 's/\-present/-2022/' $f; done

g co HEAD -- build/meson/
```
 2022-12-20 12:44:56 -05:00
W. Felix Handte
36d5c2f326 Update Copyright Year ('2021' -> 'present') 
```
for f in $(find . \( -path ./.git -o -path ./tests/fuzz/corpora -o -path ./tests/regression/data-cache -o -path ./tests/regression/cache \) -prune -o -type f);
do
  sed -i 's/\-2021/-present/' $f;
done

g co HEAD -- .github/workflows/dev-short-tests.yml # fix bad match
```
 2022-12-20 12:42:50 -05:00
W. Felix Handte
8927f985ff Update Copyright Headers 'Facebook' -> 'Meta Platforms' 
```
for f in $(find . \( -path ./.git -o -path ./tests/fuzz/corpora \) -prune -o -type f);
do
  sed -i 's/Facebook, Inc\./Meta Platforms, Inc. and affiliates./' $f;
done
```
 2022-12-20 12:37:57 -05:00
Harmen Stoppels
efef80b75e Fix make variable 2022-08-19 12:06:43 +02:00
Elliot Gorokhovsky
0c386afbfd
Fix CI failures by adding apt-get update to Github Actions (#3082) 
* Fix CI failures by adding apt-get update to Makefile

* Fix travis failure caused by apt-get update

* Move apt-get update from Makefile to Github Actions .yml

* Revert .travis.yml change

* Fix typo
 2022-03-03 14:52:06 -05:00
Yann Collet
80a28f2078 update regression results 2021-12-20 07:54:57 -08:00
Yann Collet
9fc7f38e12 added target update_regressionResults 
to automate the creation of updated tests/regression/results.csv
 2021-12-16 21:46:14 -08:00
Yann Collet
7868f38019
Merge pull request #2747 from Helflym/dev 
Add AIX support in Makefile
 2021-10-01 08:13:39 -07:00
Nick Terrell
8bf699aa59 [build] Add support for ASM files in Make + CMake 
* Extract out common portion of `lib/Makefile` into `lib/libzstd.mk`.
  Most relevantly, the way we find library files.
* Use `lib/libzstd.mk` in the other Makefiles instead of repeating the
  same code.
* Add a test `tests/test-variants.sh` that checks that the builds of
  `make -C programs allVariants` are correct, and run it in Actions.
* Adds support for ASM files in the CMake build.

The Meson build is not updated because it lists every file in zstd,
and supports ASM off the bat, so the Huffman ASM commit will just add
the ASM file to the list.

The Visual Studios build is not updated because I'm not adding ASM
support to Visual Studios yet.
 2021-09-17 14:13:53 -07:00
Yann Collet
72bd2a83a0 reduce length of scanbuild static analyzer test 
This was ~30mn, by far the longest run on travisCI.
That's because it re-analyzes multiple times the same files (library files notably).
It also performs actions that make no sense for the static analyzer purpose,
such as building the single-file library.

Reduced time spent in this test by reducing its scope :
just build the CLI, and obviously the library along it.
These are the only ones that really deserve to be analyzed.

Unfortunately, it still results in a number of false positives when using newer versions of scanbuild
(each version of scanbuild generates a different list of false positives).
These will have to be fixed before transfering to Github Actions.
 2021-08-29 15:26:31 -07:00
Clément Chigot
399849e236 Makefile: add AIX support 
For lib, AIX linker doesn't allow --soname.
 2021-08-13 10:25:14 +02:00
sen
13449d7ce1
Add PHONY targets to makefiles (#2629) 2021-05-07 14:03:19 -04:00
sen
91465e23b2
[1.5.0] Enable multithreading in lib build by default (#2584) 
* Update lib Makefile to have new targets

* Update lib/README.md for mt
 2021-05-07 11:13:30 -04:00
W. Felix Handte
4ba49af665 Rewrite References to Location 2021-05-05 18:03:48 -04:00
Yann Collet
f2c0312889 removing signed integer overflow exception from ubsan tests 2021-02-19 16:30:06 -08:00
Yann Collet
9b0772177c fix ubsan test errors 
allows recovering from pointer overflow
 2021-02-19 10:42:05 -08:00
Nick Terrell
66e811d782 [license] Update year to 2021 2021-01-04 17:53:52 -05:00
Yann Collet
f2ac2b7bcf try to fix cross-compiler tests 2020-12-21 15:43:14 -08:00
Yann Collet
585196353d fix ppc64 build on circleci 2020-12-20 21:44:40 -08:00
Yann Collet
9648bf027b try to keep libzstd.a "as is" once created 
to be compatible with scenarios such as
`make -j allmost`
 2020-12-20 17:10:57 -08:00
Yann Collet
ecffa77e12 streamlined example recipe 
better use of default rules
 2020-12-15 02:07:08 -08:00
senhuang42
a8693ddef7 Add seekable tests to CI 2020-12-03 09:25:45 -05:00
Yann Collet
01ecad2326 zstd in programs/ can also receive a DEBUGLEVEL value 
simplify tests : only set DEBUGLEVEL, not the flags directly
 2020-10-21 23:13:46 -07:00
Yann Collet
105b5839de added comment on $(Q) 2020-05-29 12:40:59 -07:00
Yann Collet
6ac6472988 fixed double-compilation of lib with make -j all 
also implemented verbose mode (make V=1)
 2020-05-29 12:09:51 -07:00
W. Felix Handte
78aa9373cb Add libzstd.pc Build to More Aggregate Targets in Makefiles 2020-05-08 16:11:32 -04:00
Bimba Shrestha
df9e5b6f4c adding 2020-05-07 22:07:40 -05:00
Nick Terrell
108a5572a5
Merge pull request #2048 from nocnokneo/ctest-support 
Add CTest support
 2020-04-28 11:01:13 -07:00
Carl Woffenden
4e6027f8ca
Fixed-up single-file libs directory 2020-04-07 13:35:19 +02:00
Carl Woffenden
7c420344d2 Single-file decoder script can now (optionally) create an encoder 
To complement the single-file decoder a new script was added to create an amalgamated single-file of all of the Zstd source, along with examples and (simple) tests.
 2020-04-03 19:07:46 +02:00
Taylor Braun-Jones
5b054d9e90 Add support for running more tests via CTest 2020-03-30 15:14:00 -04:00
Nick Terrell
ac58c8d720 Fix copyright and license lines 
* All copyright lines now have -2020 instead of -present
* All copyright lines include "Facebook, Inc"
* All licenses are now standardized

The copyright in `threading.{h,c}` is not changed because it comes from
zstdmt.

The copyright and license of `divsufsort.{h,c}` is not changed.
 2020-03-26 17:02:06 -07:00
Nick Terrell
3ed0f65158 [cmake] Add playTests.sh as a test 2020-01-13 14:16:15 -08:00
Bimba Shrestha
eb76f786bc [bench] Automated benchmarking script (#1906) 
* Initial revised automated benchmarking script

* Updating nb_iterations and making loop infinite

* Allowing benchmarking params to be changed from cli

* Renaming old speed test

* Removing numpy dependency for cli

* Change filename and benchmakr on pr level

* Moving build outside loop and adding iterations param

* Moving benchmarking to seperate travis ci test

* Fixing typo and using unused variable

* Added mode labels and updated README accordingly

* Adding new mode 'current' that compraes facebook:dev against current hash

* Typo

* Reverting previous accidental diff

* Typo

* Adding frequency config variable to prevent github from blacklisting

* Added new argument for frequency of fetching new prs

* Updating documentation
 2020-01-06 14:19:11 -08:00
Ryan Schmidt
6ddb783c3e Change c90 to c89 
c89 and c90 are the same language. Old compilers like gcc 4.2.1 don't
know the c90 alias.
 2019-11-07 11:04:31 -06:00
Yann Collet
b9b9a1c8e9 fix education decoder test 
when `zstd` in not installed on local system
by allowing `ZSTD` variable to hold a custom location for the binary
 2019-09-10 09:36:02 -07:00
Yann Collet
2b0a271ed2 fix eductional decoder 
fix #1774
also :
- fix minor compilation warnings
- make sure the `test` is run during CI tests
 2019-09-06 14:30:13 -07:00
Yann Collet
e7ad20c9dc update test script 2019-08-28 13:46:18 -07:00
Yann Collet
416157df80 try to debug issues on CirrusCI 2019-08-27 16:38:21 -07:00
Yann Collet
cd43f7d965 added single_file_decoder to contrib/ tests 2019-08-27 16:04:12 -07:00
Yann Collet
b12a5b9117 removed associated test 2019-08-27 14:52:16 -07:00
Yann Collet
2a6aa6be5f updated clang tests 
target clangbuild
 2019-01-19 23:40:41 -08:00
Yann Collet
9792acda3b Merge branch 'dev' into btultra2 2018-12-12 20:18:27 -08:00
Yann Collet
ea441ee4ae make test can accept DEBUGLEVEL argument 2018-12-09 23:20:51 -08:00
Yann Collet
7ef7dc561a check availability of --color=never command on grep and egrep 
before applying them.
Fixes #1436
 2018-12-03 15:46:55 -08:00
Yann Collet
1e6208e75e bumped version number to v1.3.7 
updated documentation
 2018-10-11 14:40:12 -07:00