1
0
mirror of https://github.com/facebook/zstd.git synced 2025-07-05 23:27:28 +02:00
Commit Graph

494 Commits

Author SHA1 Message Date
d8e215cbee created ZSTD_compress2() and ZSTD_compressStream2()
ZSTD_compress_generic() is renamed ZSTD_compressStream2().

Note that, for the time being,
the "stable" API and advanced one use different parameter planes :
setting parameters using the advanced API does not influence ZSTD_compressStream()
and using ZSTD_initCStream() does not influence parameters for ZSTD_compressStream2().
2018-11-30 11:25:56 -08:00
41c7d0b1e1 changed hashEveryLog into hashRateLog 2018-11-21 14:36:57 -08:00
e874dacc08 changed searchLength into minMatch
refactored all relevant API and calls
for consistency.
2018-11-20 14:56:07 -08:00
3b838abf97 ZSTD_CCtx_setParameter : value argument is now int
for compatibility with compression level
2018-11-20 11:53:01 -08:00
06c8d5a4f4 Merge branch 'dev' into advancedAPI
fixed rsyncable
2018-11-15 10:51:24 -08:00
4316409427 [cli] Set the job size 2018-11-14 17:00:29 -08:00
f9a671ac37 [cli] Add rsyncable 2018-11-14 17:00:16 -08:00
d7e10a774a added constant ZSTD_WINDOWLOG_LIMIT_DEFAULT
answering #1407.

Also : removed obsolete function ZSTD_setDStreamParameter()
which could only be used with one parameter (DStream_p_maxWindowSize).
Now replaced by ZSTD_DCtx_setWindowSize() (which exists since a few revisions)
2018-11-13 18:12:34 -08:00
2c8fde538f added constant ZSTD_MAGIC_SKIPPABLE_MASK
and updated several API comments
2018-11-13 17:36:35 -08:00
b83d1e7714 removed some static const variables
and replaced by traditional macro constants.

Unfortunately, C doesn't consider `static const` to mean "constant"
2018-11-13 16:56:32 -08:00
433059bbb2 fixed minor gcc warning
gcc-8 on Linux doesn't like usage of strncat :
`warning: ‘strncat’ output truncated before terminating nul copying as many bytes from a string as its length`.

Not sure what was wrong, it might be a false positive,
but the logic is simple enough to replaced by a simple `memcpy()`,
thus avoiding the shenanigans of null-terminated strings.
2018-10-10 17:06:25 -07:00
b304b679e5 use #if BACKTRACE_ENABLE directly
as suggested by @terrelln
2018-10-09 17:56:59 -07:00
0ea286f740 Merge branch 'backtrace' of github.com:facebook/zstd into backtrace 2018-10-09 17:24:48 -07:00
70d8c2a031 fileio.c: Disable backtrace when built with address sanitizer
Covers clang and gcc's sanitizer flags.
Can still be overridden through CFLAGS on commandline.
2018-10-09 17:14:57 -07:00
e0ab6b61b7 fixed explicit BACKTRACE order
and automatic linux backtrace detection :
__GLIBC__ must be tested after #include <features.h>
2018-10-09 17:12:21 -07:00
1e0c5466c5 fixed BACKTRACE_ENABLE macro test 2018-10-09 16:36:57 -07:00
f17c1df1ac backtrace support compiled with more conditions
following #1356,
only enable backtrace compilation on linux+glibc.

Also, disable backtrace by default from "release" compilation,
so that less platforms get impacted by the new requirements.
Can be manually enabled/disabled using BACKTRACE=1/0.
2018-10-08 17:03:06 -07:00
1d75697d9b don't assume __linux__ means __GLIBC__ on BACKTRACES_ENABLE
check for __GLIBC__ instead of __linux__ as musl libc doesn't provide
execinfo.h

fixes compilation on Alpine Linux and Void Linux musl arches.
2018-10-05 21:23:45 -03:00
b1407f9acd fixed wrong assert() position
could fire on invalid input.
blocking for afl tests.
2018-10-03 12:43:59 -07:00
3ca6261223 fixed static analyzer warnings
note : for some reason,
scan-build version on my laptop found problems within fastcover.c
that scan-build on travisCI does not flag.

They are, as usual, false positive :
the analyzer does not understand that a table (`offset`) is correctly filled before usage.
2018-10-02 15:59:11 -07:00
9012b6cba0 ./zstd -f do no longer overwrite destination file
if source file does not exist (#1082)
2018-10-01 17:16:34 -07:00
c7bd6a41ab zstd -d -f do no longer erase destination file
when source file does not exist (#1082)
2018-10-01 14:04:00 -07:00
1ab71a8e72 regroup name creation logic into its own function
for a cleaner main file decompression loop
2018-09-28 18:19:23 -07:00
ec1cb8e996 changed macro name
from EXIT_IF() to RETURN_IF()

EXIT could be misunderstood as exit(), which terminates program execution.
But the macro only leaves the function, not the program.
2018-09-28 16:04:00 -07:00
05c0a072b7 minor improvement in the multi-format suffix selection 2018-09-28 15:57:35 -07:00
d987ab5983 fixed unreachable section warning on Visual 2018-09-28 09:34:16 -07:00
913a0365b6 Merge branch 'dev' into donotdelete 2018-09-27 19:23:55 -07:00
ef1272737b fixed minor Visual conversion warnings 2018-09-27 18:29:15 -07:00
9b45db7fa6 minor refactoring of --list
trying to reduce recurrent patterns.
2018-09-27 16:49:08 -07:00
f2d6db45cd [zstd] Add -Wmissing-prototypes 2018-09-27 15:24:48 -07:00
6c51bf420c bounds for --adapt mode
can supply min and max compression level through advanced command :
--adapt=min=#,max=#
2018-09-25 16:03:28 -07:00
04f47bbdd2 Merge branch 'dev' into adapt 2018-09-24 16:56:45 -07:00
0250ac74ce fixed minor scan-build warnings 2018-09-24 00:52:19 -07:00
00c18c0c88 simplified "slows down when compression blocked" 2018-09-21 16:35:43 -07:00
c484345a82 Merge branch 'mingw' into adapt 2018-09-21 16:00:46 -07:00
bfff4f4809 ensure all writes to job->cSize are mutex protected
even when reporting errors,
using a macro for code brevity, as suggested by @terrelln,
2018-09-21 16:00:39 -07:00
54001f3dbc fix mingw compatibility
only enable backtraces for platforms we know support it
aka mac OS-X and Linux.
can be extended later.
2018-09-21 14:46:09 -07:00
ca02ebee07 removed static variables
so that --adapt can work on multiple input files too
2018-09-19 15:25:50 -07:00
89bc309d90 error out when --adapt is associated with --single-thread
since they are not compatible
2018-09-19 14:49:13 -07:00
2f78228f65 Merge branch 'dev' into adapt 2018-09-19 12:43:42 -07:00
b9118ecdab Update comments, and LD flag usage in Make 2018-09-11 14:49:47 -07:00
d4337b6f1d Move ABRThandler func out of internal lib 2018-09-11 11:39:49 -07:00
a06574fc97 Print a stack trace on unexpected term signal (e.g. SIGABRT)
For OSX and Linux, add a signal handler to SIGABRT, SGIFPE, SIGILL,
SIGSEGV, and SIGBUS. When the program terminates unexpectedly the
handler will print the current stack to the terminal to help determine
the location of the failure.

On OSX the output will look like:

```
Stack trace:
4   zstd                                0x000000010927ed96 main + 16886
5   libdyld.dylib                       0x00007fff767d1015 start + 1
6   ???                                 0x0000000000000001 0x0 + 1
```

On Linux the output will look like:

```
Stack trace:
./zstd() [0x4b8e1b]
./zstd() [0x4b928a]
./zstd() [0x403dc2]
/lib64/libc.so.6(__libc_start_main+0xf5) [0x7f5e0fbb0445]
./zstd() [0x405754]
```

As is, the code does not function on WIN32.

See also: https://oroboro.com/stack-trace-on-crash/
2018-09-06 18:46:52 -07:00
78af534f82 Fixed unsafe string copy and concat in fileio.c.
Per warnings from flawfinder: "Does not check for buffer overflows when
copying to destination [MS-banned] (CWE-120). Consider using snprintf,
strcpy_s, or strlcpy (warning: strncpy easily misused).".

Replaced called to strcpy and strcat in `fileio.c` to calls with a
specified size (`strncpy` and `strncat`).

Tested the changes on OSX, Linux, Windows.
On OSX + Linux, changes were tested with ASAN. The following flags were
used: 'check_initialization_order=1:strict_init_order=1:detect_odr_violation=1:detect_stack_use_after_return=1'

To reproduce warning:
./flawfinder.py ./programs/fileio.c
2018-08-20 22:15:24 -04:00
105677c6db created ZSTDMT_toFlushNow()
tells in a non-blocking way if there is something ready to flush right now.
only works with multi-threading for the time being.

Useful to know if flush speed will be limited by lack of production.
2018-08-17 18:11:54 -07:00
09e63c58ac fix : no longer slow down on input saturation
only slows down when all buffers are full
2018-08-17 16:27:43 -07:00
8b674d7dc7 ensured compression level is maxed at ZSTD_maxCLevel() 2018-08-17 16:01:56 -07:00
3e4617ef54 frameProgression reports nbActiveWorkers and output flushed 2018-08-14 11:49:25 -07:00
0853f86044 adaptive mode uses default window size of 8 MB 2018-08-13 13:13:22 -07:00
33f7709c71 fileio: changed parameter type from ptr to plain structure
safer : this parameter is read-only,
we don't want original structure to be modified
2018-08-13 13:02:03 -07:00