1
0
mirror of https://github.com/facebook/zstd.git synced 2025-03-07 09:26:03 +02:00

9136 Commits

Author SHA1 Message Date
W. Felix Handte
7c24c3e6ce Give Up on Searching End of Block
Amusingly, it seems to be a non-trivial performance hit to add in final
searches or even hash table insertions during cleanup. So let's not. It seems
to not make any meaningful difference in compression ratio.
2021-09-01 14:15:03 -04:00
W. Felix Handte
35932ab2f1 Prefetch Input in Incompressible Sections (+0.25% Speed) 2021-09-01 14:15:03 -04:00
W. Felix Handte
b092dd75b7 Shrink Pipeline from 4 Positions to 3 2021-09-01 14:15:03 -04:00
W. Felix Handte
387840af79 Re-Order Operations for Slightly Better Performance 2021-09-01 14:15:03 -04:00
W. Felix Handte
bc768bccc0 Track Step Size Statefully, Rather than Recalculating Every Time 2021-09-01 14:15:03 -04:00
W. Felix Handte
80bc12b33a Initial Pipelined Implementation for ZSTD_fast 2021-09-01 14:15:03 -04:00
W. Felix Handte
ab8aa49b8d Fix Benchmark Corruption Display 2021-09-01 14:15:03 -04:00
Yann Collet
6715096611
Merge pull request #2758 from facebook/qemu
added qemu tests
2021-08-31 09:56:50 -07:00
Yann Collet
c1de65535f Merge branch 'dev' into qemu 2021-08-31 08:16:46 -07:00
Yann Collet
6933ac67d3
Merge pull request #2757 from facebook/transferGA
Reduce test time on TravisCI
2021-08-31 07:40:21 -07:00
Yann Collet
333ecf6865 add powerpc qemu emulation 2021-08-30 06:37:50 -07:00
Yann Collet
2b27d07d06 attempt at adding m68k qemu tests
with optional success (for the time being)
2021-08-29 21:39:06 -07:00
Yann Collet
1e5c90cb5b remove qemu tests
that are being transfered to GA in #2758.
This represents a saving of ~25mn of cpu time on TravisCI.
2021-08-29 20:54:18 -07:00
Yann Collet
74b4171fb8 fix alignment condition in FSE_buildCTable
2-bytes alignment is enough for 16-bit fields
2021-08-29 19:05:04 -07:00
Yann Collet
f21977c5e6 fix playTests.sh when EXE_PREFIX not null 2021-08-29 17:20:12 -07:00
Yann Collet
18191c85c9 adding optional QEMU_SYS 2021-08-29 16:43:32 -07:00
Yann Collet
1c97ec73d7 added qemu tests
running zstd library on emulated targets
2021-08-29 16:28:41 -07:00
Yann Collet
b341aa2f95 remove versions-compatibility test from GA
since it fails on Github Actions specifically.

The test is run on TravisCI for the time being.
Its duration has been reduced to ~6mn anyway.
2021-08-29 15:47:04 -07:00
Yann Collet
72bd2a83a0 reduce length of scanbuild static analyzer test
This was ~30mn, by far the longest run on travisCI.
That's because it re-analyzes multiple times the same files (library files notably).
It also performs actions that make no sense for the static analyzer purpose,
such as building the single-file library.

Reduced time spent in this test by reducing its scope :
just build the CLI, and obviously the library along it.
These are the only ones that really deserve to be analyzed.

Unfortunately, it still results in a number of false positives when using newer versions of scanbuild
(each version of scanbuild generates a different list of false positives).
These will have to be fixed before transfering to Github Actions.
2021-08-29 15:26:31 -07:00
Yann Collet
7f37b8a547 accelerate versionsCompatibilityTest
by allowing parallel build of units,
and reducing optimization levels.

Parallel build is only effective on "recent" versions of `zstd`,
as previously, the list of units was passed as a list of source files,
which is something neither `make` nor `gcc` can parallelize.
So its impact is mildly effective (-20%).

Reducing optimization level to `-O1` makes compilation much faster.
It also makes runtime slower,
but in this test, compilation time dominates run time.
The savings are very significant (-50%).

On my test system, it reduces the length of this test from 13mn to 5mn.
2021-08-29 14:48:11 -07:00
Yann Collet
ef69539849 transferred inter-versions compatibility tests to GA 2021-08-29 11:53:56 -07:00
sen
1f3fc1936c
Merge pull request #2753 from senhuang42/better_error_msg
[easy] Fix zstd bench error message
2021-08-23 20:37:48 -04:00
senhuang42
dce48f53df Fix benchzstd error message 2021-08-23 19:10:16 -04:00
Yann Collet
be82a0ab8f
Merge pull request #2746 from eli-schwartz/meson-fixup
meson fixups
2021-08-23 15:57:47 -07:00
Yann Collet
18a20b3ad7
Merge pull request #2752 from facebook/hashLog3max
make ZSTD_HASHLOG3_MAX private
2021-08-20 12:51:17 -07:00
Yann Collet
2de42174bb make ZSTD_HASHLOG3_MAX private
This is an implementation detail,
it doesn't belong to public space (zstd.h).
2021-08-20 09:52:42 -07:00
sen
ae998544de
Merge pull request #2750 from senhuang42/sb_compress
Improve branch misses on FSE symbol spreading
2021-08-20 12:47:24 -04:00
senhuang42
da095ed899 Improve branch misses on FSE symbol spreading 2021-08-18 10:22:22 -07:00
Clément Chigot
6ef6cd7999 test: avoid /dev/full on AIX 2021-08-13 10:25:50 +02:00
Clément Chigot
399849e236 Makefile: add AIX support
For lib, AIX linker doesn't allow --soname.
2021-08-13 10:25:14 +02:00
Eli Schwartz
9748608aeb
meson: set the symbol visibility of the shared library to hidden
This matches the Makefile build. Due to one private xxhash symbol in use
by the program, it recompiles a private copy of xxhash.

Due to the test binaries making extensive (?) use of private symbols, it
doesn't even attempt to link to shared libzstd, and instead, all of the
original object files are added to libtestcommon itself for private
linkage. This, too, matches the Makefile build.

Ref. #2261
2021-08-09 22:03:17 -04:00
Eli Schwartz
d95a3f5097
meson: fix warnings in build files
meson prefers that project-level options for Wall/Wextra/pedantic be
used, rather than hardcoding raw flags in add_project_arguments. If you
do the latter anyway, it raises a meson warning.

Set the default options for the project to use all this.

Also move the -Werror comment to the project default options with
appropriate format, but leave it commented out since it does not work.
2021-08-09 20:17:00 -04:00
Eli Schwartz
ee3355df30
meson: link tests with a convenience library of sources used by multiple binaries
This saves repeatedly compiling them, resulting in 16 fewer compile
steps, in exchange for one `ar` step.
2021-08-09 20:15:44 -04:00
sen
6a258043f5
Merge pull request #2692 from senhuang42/rebalance_clevel
[RFC] Rebalance compression levels
2021-08-06 12:51:31 -04:00
Sen Huang
539b3aab9b Optimize 32-bit VecMask_next() 2021-08-04 17:14:58 -04:00
Felix Handte
ae131282f5
Merge pull request #2742 from felixhandte/set-mtime
Set mtime on Output Files
2021-08-04 16:44:46 -04:00
senhuang42
e411040ea1 Add 64 row entry support for lazy 2021-08-04 16:19:12 -04:00
senhuang42
31820e032c Rebalance clevels for lazy 2021-08-04 16:18:52 -04:00
sen
f9b034086a
Merge pull request #2732 from senhuang42/huf_sort_improvement
[HUF] Improve Huffman sorting algorithm
2021-08-04 16:07:56 -04:00
W. Felix Handte
a8f4612eab Remove sleep()s in Test; Replace with Artificial mtime
This behavior of `touch` is standardized in POSIX, so it should be available.
2021-08-04 15:56:46 -04:00
W. Felix Handte
9cd6c1ff4d Update mtime and atime for Written Files 2021-08-04 14:49:56 -04:00
W. Felix Handte
a719edbbc3 Pull utime() Call into Helper 2021-08-04 14:49:00 -04:00
senhuang42
aa1957477b Improve Huffman sorting algorithm 2021-08-04 12:43:34 -04:00
W. Felix Handte
ead41bcb4e Add Test to Verify mtime is Copied to Destination 2021-08-03 17:22:58 -04:00
Nick Terrell
6ee70bae46
Merge pull request #2733 from terrelln/huf-cspeed
[HUF] Improve Huffman encoding speed
2021-08-03 12:59:54 -04:00
Nick Terrell
d8a0797268 [fuzz] Add Huffman round trip fuzzer
* Add a Huffman round trip fuzzer
* Fix two minor bugs in Huffman that aren't exposed in zstd
  - Incorrect weight comparison (weights are allowed to be equal to
    table log).
  - HUF_compress1X_usingCTable_internal() can return compressed
    size >= source size, so the assert that `cSize <= 65535` isn't
    correct, and it needs to be checked instead.
2021-08-03 08:10:06 -07:00
sen
5c46f62006
Merge pull request #2677 from senhuang42/ci_overhaul_2
[CI][2/2] Migrate CI tests which (currently) fail
2021-08-02 09:55:49 -04:00
Sen Huang
5ec7897a26 Fix static analyzer warnings 2021-07-29 09:11:12 -07:00
Sen Huang
43aad811db Add VS2019 test, migrate minimal decompressor macros to GHActions 2021-07-29 08:49:36 -07:00
Nick Terrell
32cfe15edc
Merge pull request #2724 from jonringer/fix-pzstd-static-build
pzstd: fix linking for static builds
2021-07-28 14:53:01 -07:00