1
0
mirror of https://github.com/facebook/zstd.git synced 2025-03-06 16:56:49 +02:00

2215 Commits

Author SHA1 Message Date
W. Felix Handte
11aba9b316 make man 2022-12-22 14:13:24 -05:00
W. Felix Handte
382026f096 Man Page Tweaks, Edits, Formatting Fixes
This started as an application of the edits suggested in #3201 and expanded
from there.
2022-12-22 14:13:17 -05:00
Nick Terrell
40a7188130 Fix make clangbuild & add CI
Fix the errors for:
* `-Wdocumentation`
* `-Wconversion` except `-Wsign-conversion`
2022-12-21 17:31:04 -08:00
Nick Terrell
b6e8112261 [util] Add traces enabled by --trace-file-stat
Print traces to stderr when --trace-file-stat is passed.
We trace all functions that read and write file metadata.
2022-12-21 17:19:34 -08:00
Yann Collet
7607b96ed9
Merge pull request #3385 from jonpalmisc/improve_help
Improve help/usage (`-h`, `-H`) formatting
2022-12-21 10:56:09 -08:00
Jon Palmisciano
9c93dd71cd
Clean up welcome message 2022-12-20 20:13:00 -05:00
Jon Palmisciano
678335c4f3
Rewrite help output to improve readability 2022-12-20 20:12:59 -05:00
W. Felix Handte
5d693cc38c Coalesce Almost All Copyright Notices to Standard Phrasing
```
for f in $(find . \( -path ./.git -o -path ./tests/fuzz/corpora -o -path ./tests/regression/data-cache -o -path ./tests/regression/cache \) -prune -o -type f); do sed -i '/Copyright .* \(Yann Collet\)\|\(Meta Platforms\)/ s/Copyright .*/Copyright (c) Meta Platforms, Inc. and affiliates./' $f; done

git checkout HEAD -- build/VS2010/libzstd-dll/libzstd-dll.rc build/VS2010/zstd/zstd.rc tests/test-license.py contrib/linux-kernel/test/include/linux/xxhash.h examples/streaming_compression_thread_pool.c lib/legacy/zstd_v0*.c lib/legacy/zstd_v0*.h
nano ./programs/windres/zstd.rc
nano ./build/VS2010/zstd/zstd.rc
nano ./build/VS2010/libzstd-dll/libzstd-dll.rc
```
2022-12-20 12:52:34 -05:00
W. Felix Handte
7f12f24cf4 Rewrite Copyright Date Ranges from -present to -2022
Apparently it's better. Somehow.

```
for f in $(find . \( -path ./.git -o -path ./tests/fuzz/corpora -o -path ./tests/regression/data-cache -o -path ./tests/regression/cache \) -prune -o -type f); do echo $f; sed -i 's/\-present/-2022/' $f; done

g co HEAD -- build/meson/
```
2022-12-20 12:44:56 -05:00
W. Felix Handte
8927f985ff Update Copyright Headers 'Facebook' -> 'Meta Platforms'
```
for f in $(find . \( -path ./.git -o -path ./tests/fuzz/corpora \) -prune -o -type f);
do
  sed -i 's/Facebook, Inc\./Meta Platforms, Inc. and affiliates./' $f;
done
```
2022-12-20 12:37:57 -05:00
daniellerozenblit
e2fc93340f
Merge branch 'dev' into http-to-https 2022-12-15 10:46:13 -05:00
Nick Terrell
15f32ad74c [fileio] Separate parameter adaption from display update rate
Split the logic for parameter adaption from the logic to update the display rate.
This decouples the two updates, so changes to display updates don't affect
parameter adaption.

Also add a test case that checks that parameter adaption actually happens.

This fixes Issue #3353, where --adapt is broken when --no-progress is passed.
2022-12-14 17:08:21 -08:00
Nick Terrell
fbff7827fa Refactor progress bar & summary line logic
* Centralize the logic about whether to print the progress bar or not in
  the `*_PROGRESS()` macros.
* Centralize the logc about whether to print the summary line or not in
  `FIO_shouldDisplayFileSummary()` and
  `FIO_shouldDisplayMultipleFileSummary()`.
* Make `--progress` work for non-zstd (de)compressors.
* Clean up several edge cases in compression and decompression progress
  printing along the way. E.g. wrong log level, or missing summary line.

One thing I don't like about stdout mode, which sets the display level
to 1, is that warnings aren't displayed. After this PR, we could change
stdout mode from lowering the display level, to defaulting to implied
`--no-progress`. But, I think that deserves a separate PR.
2022-12-14 15:44:09 -08:00
Nick Terrell
e58a39f84e Allow tests to fake stdin/stdout/stderr is a console
We've been unable to effectively test cases where stdin/stdout/stderr
are consoles, because in our test cases they generally aren't. Allow the
command line flags `--fake-std{in,out,err}-is-console` to tell the CLI
to pretend that std{in,out,err} is a console.
2022-12-14 15:44:09 -08:00
Danielle Rozenblit
4dffc35f2e Convert references to https from http 2022-12-14 06:58:35 -08:00
FRex
dc39409a03 Print checksum value for single frame files in cli with -v -l options 2022-12-12 10:53:57 -08:00
Yann Collet
4d82a4d3f2 minor : reorder --help
more logical context grouping
2022-11-22 07:23:45 -08:00
Elliot Gorokhovsky
b1182fe242
Merge pull request #3300 from yiyuaner/dev
Fix the wrong check for buffer overrun in UTIL_mergeFileNamesTable
2022-11-02 15:28:02 -04:00
appleman4321
5334339421 Rewrite checking condition bufStart pointer NULL 2022-11-01 02:31:09 +03:00
yiyuaner
361d86998a Fix the wrong check for buffer overrun in UTIL_mergeFileNamesTable 2022-10-24 20:37:15 +08:00
Yann Collet
9abecfbb7e replaced Zstandard by zstd in man page
identified by @paulmenzel in #3286
2022-10-11 23:32:18 -07:00
Ilya Kurdyukov
2bd70eff06
zstd-pgo: also clean zstd binary from objects
Just a precaution, because it works anyway.
2022-10-06 00:38:23 +07:00
Ilya Kurdyukov
2ffcb2d6a8
fixed zstd-pgo target for GCC
Since your Makefile uses obj/$(HASH_DIR) for object files, this code does not work correctly for GCC. Because profiles are saved in one directory, and are expected in another when reading.

`$(RM) zstd *.o` - this line doesn't delete object files.

Clang stores profiles in the current directory, so the problem doesn't appear when compiling with Clang.

Also this code will work if BUILD_DIR is set.
2022-10-05 17:32:16 +07:00
ctkhanhly
358787764f
Fix for zstd CLI accepts bogus values for numeric parameters (#3268)
* add checks to mal-formed numeric values for memory and memlimit parameters

Signed-off-by: Ly Cao <lycao@fb.com>

* changed errorMsg to a literal string instead of static string in main

* moved bogus numeric error to NEXT_UINT32 + add macro NEXT_TSIZE

Signed-off-by: Ly Cao <lycao@fb.com>

Signed-off-by: Ly Cao <lycao@fb.com>
Co-authored-by: Ly Cao <lycao@fb.com>
2022-09-21 13:20:01 -07:00
Yann Collet
f7251f88b9
Merge pull request #3247 from haampie/fix/grep
Fix make variable
2022-09-19 10:47:38 -07:00
Yann Collet
5129b4ab10 simplify clean target maintenance within programs/ 2022-09-12 12:19:35 -07:00
Harmen Stoppels
efef80b75e Fix make variable 2022-08-19 12:06:43 +02:00
Chris Burgess
3b4e47092e
Document pass-through behavior (#3242)
Adds documentation to help and man pages for legacy pass-through behavior
when force is set and destination is stdout. Documents --pass-through in
man pages
2022-08-15 10:29:54 -07:00
Nick Terrell
03cc84fddb
Add explicit --pass-through flag and default to enabled for *cat (#3223)
Fixes #3211.

Adds the `--[no-]pass-through` flag which enables/disables pass-through mode.

* `zstdcat`, `zcat`, and `gzcat` default to `--pass-through`.
  Pass-through mode can be disabled by passing `--no-pass-through`.
* All other binaries default to not setting pass-through mode.
  However, we preserve the legacy behavior of enabling pass-through
  mode when writing to stdout with `-f` set, unless pass-through
  mode is explicitly disabled with `--no-pass-through`.

Adds a new test for this behavior that should codify the behavior we want.
2022-08-04 17:15:59 -07:00
zengyijing
d0dcc9d775
fix issue #3144 (#3226)
* fix issue #3144

* add test case for verbose-wlog

Co-authored-by: zengyijing <yijingzeng@fb.com>
2022-08-04 13:51:14 -07:00
Yann Collet
e818fa8eb0
Merge pull request #3232 from facebook/fileiotypes_nomemh
fileio_types.h : avoid dependency on mem.h
2022-08-03 22:57:16 +02:00
Yann Collet
3dfcafacd7 fileio_types.h : avoid dependency on mem.h
fileio_types.h cannot be parsed by itself
because it relies on basic types defined in `lib/common/mem.h`.
As for #3231, it likely wasn't detected because `mem.h` was probably included before within target files.
But this is not proper.

A "easy" solution would be to add the missing include,
but each dependency should be considered "bad" by default,
and only allowed if it brings some tangible value.

In this case, since these types are only used to declare internal structure variables
which are effectively only flags,
I believe it's really not valuable to add a dependency on `mem.h` for this purpose
while the standard `int` type can do the same job.

I was expecting some compiler warnings following this change,
but it turns out we don't use `-Wconversion` by default on `zstd` source code,
so there is none.

Nevertheless, I enabled `-Wconversion` locally and proceeded to fix a few conversion warnings in the process.

Adding `-Wconversion` to the list of flags used for `zstd` is something I would be favorable over the long term,
but it cannot be done overnight,
because the nb of places where this warning is triggered is daunting.
Better progressively reduce the nb of triggered `-Wconversion` warnings before enabling this flag by default.
2022-08-03 21:39:35 +02:00
Yann Collet
a925362534 minor : fixed missing include
I presume it was not detected before
because "fileio.h" is probably always included after "util.h".
2022-08-03 20:52:15 +02:00
Elliot Gorokhovsky
28ceb63503
Merge pull request #3220 from embg/issue3200
Disallow empty string as argument for --output-dir-flat and --output-dir-mirror
2022-08-01 14:04:57 -04:00
Yonatan Komornik
ae4670466c
stdin multiple file fixes (#3222)
* Fixes for https://github.com/facebook/zstd/issues/3206 - bugs when handling stdin as part of multiple files.

* new line at end of multiple-files.sh
2022-07-29 16:13:07 -07:00
Elliot Gorokhovsky
f9f27de91c Disallow empty output directory 2022-07-29 14:48:33 -07:00
Tom Wang
d4a5bc4efc
Add warning when multi-thread decompression is requested (#3208)
When user pass in argument for both decompression and multi-thread, print a warning message
to indicate that multi-threaded decompression is not supported.

* Add warning when multi-thread decompression is requested
* add test case for multi-threaded decoding warning
   Expectation is for -d -T0 we will not throw any warning,
   and see warning for any other -d -T(>1) inputs
2022-07-29 12:51:58 -07:00
Chris Burgess
2b9fde932b
Fix small file passthrough (#3215) 2022-07-29 12:22:46 -07:00
Qiongsi Wu
b1bbb0eb4c
[AIX] Fix Compiler Flags and Bugs on AIX to Pass All Tests (#3219)
* Fixing compiler warnings

* Replace the old -s flag with the -Wl,-s flag

* Fixing compiler warnings

* Fixing the linker strip flag and tests/code not working as expected on AIX
2022-07-29 12:21:59 -07:00
Elliot Gorokhovsky
e1873ad576 Fix buffer underflow for null dir1 2022-07-29 11:10:47 -07:00
Elliot Gorokhovsky
6d75b36b7f Clarify -B docstring 2022-07-14 00:22:21 -04:00
htnhan
d7eb829af5 Detect multiple dictIDs in one file 2022-07-08 12:20:50 -05:00
htnhan
cc8c98485a zstd -lv <file> to show dictID 2022-07-05 21:28:33 -05:00
Elliot Gorokhovsky
bb3839a78c make -C programs zstd.1 2022-06-29 14:55:14 -04:00
Yann Collet
574ecbb0fc display a warning message when using C90 clock_t for MT speed measurements. 2022-06-19 11:38:06 -07:00
Yann Collet
27bf96e72b updated --single-thread man 2022-06-07 17:45:15 -07:00
Talha Khan
14894d63c1 Typo in man 2022-05-20 16:53:48 +08:00
Yann Collet
eb726c6a20 updated man pages
had to run the conversion script on Ubuntu, as it doesn't run correctly on macos anymore.
2022-04-13 18:57:27 -07:00
Yann Collet
0df2fd6088 updated man page, providing more details for --train mode
following questions from #3111.

Note : only the source markdown has been updated,
the actual man page zstd.1 still need to be processed.
2022-04-13 18:51:59 -07:00
Yann Collet
460780f804
Merge pull request #3094 from dirkmueller/usage_cleanup
Split help in long and short version, cleanup formatting
2022-04-05 07:09:54 -07:00