1
0
mirror of https://github.com/facebook/zstd.git synced 2025-03-07 09:26:03 +02:00

2240 Commits

Author SHA1 Message Date
Yann Collet
af09777b24 ensure that benchmark mode can only be invoked with zstd format
fix #3463
2023-01-31 09:04:29 -08:00
Yann Collet
82ca00811a change logic when stderr is not console : don't update progress status
but keep warnings and final operation statement.

updated tests/cli-tests/ accordingly
2023-01-26 13:00:52 -08:00
Yonatan Komornik
7b3f03bc9d
Merge pull request #3457 from yoniko/fix-rowhash-cli
[Bugfix] CLI row hash flags set the wrong values


`--[no-]row-match-finder` do the opposite of what they are supposed to.
In effect the no option would activate row hash while the other option will disable it.
This commit fixes the issue and changes the code to use the more readable enum values.
2023-01-25 22:40:25 -08:00
Yonatan Komornik
6422d1d7a8 Bugfix: --[no-]row-match-finder do the opposite of what they are supposed to 2023-01-25 17:59:35 -08:00
Yann Collet
02434e0867 enforce a hard fail when input files are set to be erased
in scenarios where it's supposed to not be possible.

suggested by @terrelln
2023-01-25 16:18:20 -08:00
Yann Collet
8c85b29e32 disable --rm on -o command
make it more similar to -c (aka `stdout`) convention.
2023-01-25 16:09:25 -08:00
Yann Collet
b6fd91ba84 update man 2023-01-20 18:07:55 -08:00
Yann Collet
cee6bec9fa refactor : --rm is ignored with stdout
`zstd` CLI has progressively moved to the policy of
ignoring `--rm` command when the output is `stdout`.
The primary drive is to feature a behavior more consistent with `gzip`,
when `--rm` is the default, but is also ignored when output is `stdout`.
Other policies are certainly possible, but would break from this `gzip` convention.

The new policy was inconsistenly enforced, depending on the exact list of commands.
For example, it was possible to circumvent it by using `-c --rm` in this order,
which would re-establish source removal.

- Update the CLI so that it necessarily catch these situations and ensure that `--rm` is always disabled when output is `stdout`.
- Added a warning message in this case (for verbosity 3 `-v`).
- Added an `assert()`, which controls that `--rm` is no longer active with `stdout`
- Added tests, which control the behavior, even when `--rm` is added after `-c`
- Removed some legacy code which where trying to apply a specific policy for the `stdout` + `--rm` case, which is no longer possible
2023-01-20 18:04:55 -08:00
Felix Handte
3d25502c2d
Merge pull request #3432 from felixhandte/fix-perms
Fix CLI Handling of Permissions and Ownership (Again)
2023-01-20 19:19:05 -05:00
Felix Handte
772229afd5
Merge pull request #3442 from felixhandte/pgo-tests
Test PGO Builds
2023-01-20 19:18:51 -05:00
W. Felix Handte
87e169d05d Add Additional Flags to PGO Build
In GCC, we can add a couple more flags to give us confidence that the profile
data is actually being found and used.

Also, my system for example doesn't have a binary installed under the name
`llvm-profdata`, but it does have, e.g., `llvm-profdata-13`, etc. So this
commit adds a variable that can be overridden.
2023-01-20 17:32:49 -05:00
Yann Collet
ea684c335a added c89 build test to CI 2023-01-19 14:59:30 -08:00
Yann Collet
bbe65d760c
Merge pull request #3423 from facebook/ptime
Refactor timefn, restore support for clock_gettime()
2023-01-18 13:27:42 -08:00
W. Felix Handte
7a8c8f3fe7 Easy: Print Mode as Octal in chmod() Trace 2023-01-18 11:57:54 -08:00
W. Felix Handte
0d2d460223 Mimic gzip chown(gid), chmod(), chown(uid) Behavior
Avoids a race condition in which we unintentionally open up permissions to
the wrong group.
2023-01-18 11:57:54 -08:00
W. Felix Handte
1e3eba65a6 Copy Permissions from Source File 2023-01-18 11:57:35 -08:00
W. Felix Handte
0382076af7 Re-Use stat_t in FIO_compressFilename_srcFile() 2023-01-18 11:33:07 -08:00
W. Felix Handte
a5ed28f1fb Use Existing Src File Stat in *_dstFile() Funcs
One fewer `stat()` call to make per operation!
2023-01-17 14:08:22 -08:00
W. Felix Handte
5653f96776 Pass stat_t into *_dstFile() Functions 2023-01-17 14:05:15 -08:00
W. Felix Handte
2ad6855ac1 FIO_openSrcFile() Returns File Stat 2023-01-17 14:01:06 -08:00
Elliot Gorokhovsky
5d8cfa6b96
Deprecate advanced streaming functions (#3408)
* deprecate advanced streaming functions

* remove internal usage of the deprecated functions

* nit

* suppress warnings in tests/zstreamtest.c

* purge ZSTD_initDStream_usingDict

* nits

* c90 compat

* zstreamtest.c already disables deprecation warnings!

* fix initDStream() return value

* fix typo

* wasn't able to import private symbol properly, this commit works around that

* new strategy for zbuff

* undo zbuff deprecation warning changes

* move ZSTD_DISABLE_DEPRECATE_WARNINGS from .h to .c
2023-01-13 14:51:47 -05:00
Yann Collet
a2ef23dec0 restore support of clock_gettime() for POSIX systems
This should notably allow posix systems with timespec_get()
to have access to a high resolution timer,
instead of falling back to C90's clock_t.
2023-01-12 21:08:15 -08:00
Yann Collet
bcfb7ad03c refactor timefn
The timer storage type is no longer dependent on OS.
This will make it possible to re-enable posix precise timers
since the timer storage type will no longer be sensible to #include order.
See #3168 for details of pbs of previous interface.

Suggestion by @terrelln
2023-01-12 19:24:31 -08:00
Yann Collet
8b130009e3 minor simplification refactoring for timefn
`UTIL_getSpanTimeMicro()` can be factored in a generic way,
reducing OS-dependent code.
2023-01-06 16:12:54 -08:00
Yann Collet
90597d78ea
Merge pull request #3394 from terrelln/issue-3010
[cli-tests] Test file stat read/write
2022-12-27 16:20:05 -08:00
W. Felix Handte
11aba9b316 make man 2022-12-22 14:13:24 -05:00
W. Felix Handte
382026f096 Man Page Tweaks, Edits, Formatting Fixes
This started as an application of the edits suggested in #3201 and expanded
from there.
2022-12-22 14:13:17 -05:00
Nick Terrell
40a7188130 Fix make clangbuild & add CI
Fix the errors for:
* `-Wdocumentation`
* `-Wconversion` except `-Wsign-conversion`
2022-12-21 17:31:04 -08:00
Nick Terrell
b6e8112261 [util] Add traces enabled by --trace-file-stat
Print traces to stderr when --trace-file-stat is passed.
We trace all functions that read and write file metadata.
2022-12-21 17:19:34 -08:00
Yann Collet
7607b96ed9
Merge pull request #3385 from jonpalmisc/improve_help
Improve help/usage (`-h`, `-H`) formatting
2022-12-21 10:56:09 -08:00
Jon Palmisciano
9c93dd71cd
Clean up welcome message 2022-12-20 20:13:00 -05:00
Jon Palmisciano
678335c4f3
Rewrite help output to improve readability 2022-12-20 20:12:59 -05:00
W. Felix Handte
5d693cc38c Coalesce Almost All Copyright Notices to Standard Phrasing
```
for f in $(find . \( -path ./.git -o -path ./tests/fuzz/corpora -o -path ./tests/regression/data-cache -o -path ./tests/regression/cache \) -prune -o -type f); do sed -i '/Copyright .* \(Yann Collet\)\|\(Meta Platforms\)/ s/Copyright .*/Copyright (c) Meta Platforms, Inc. and affiliates./' $f; done

git checkout HEAD -- build/VS2010/libzstd-dll/libzstd-dll.rc build/VS2010/zstd/zstd.rc tests/test-license.py contrib/linux-kernel/test/include/linux/xxhash.h examples/streaming_compression_thread_pool.c lib/legacy/zstd_v0*.c lib/legacy/zstd_v0*.h
nano ./programs/windres/zstd.rc
nano ./build/VS2010/zstd/zstd.rc
nano ./build/VS2010/libzstd-dll/libzstd-dll.rc
```
2022-12-20 12:52:34 -05:00
W. Felix Handte
7f12f24cf4 Rewrite Copyright Date Ranges from -present to -2022
Apparently it's better. Somehow.

```
for f in $(find . \( -path ./.git -o -path ./tests/fuzz/corpora -o -path ./tests/regression/data-cache -o -path ./tests/regression/cache \) -prune -o -type f); do echo $f; sed -i 's/\-present/-2022/' $f; done

g co HEAD -- build/meson/
```
2022-12-20 12:44:56 -05:00
W. Felix Handte
8927f985ff Update Copyright Headers 'Facebook' -> 'Meta Platforms'
```
for f in $(find . \( -path ./.git -o -path ./tests/fuzz/corpora \) -prune -o -type f);
do
  sed -i 's/Facebook, Inc\./Meta Platforms, Inc. and affiliates./' $f;
done
```
2022-12-20 12:37:57 -05:00
daniellerozenblit
e2fc93340f
Merge branch 'dev' into http-to-https 2022-12-15 10:46:13 -05:00
Nick Terrell
15f32ad74c [fileio] Separate parameter adaption from display update rate
Split the logic for parameter adaption from the logic to update the display rate.
This decouples the two updates, so changes to display updates don't affect
parameter adaption.

Also add a test case that checks that parameter adaption actually happens.

This fixes Issue #3353, where --adapt is broken when --no-progress is passed.
2022-12-14 17:08:21 -08:00
Nick Terrell
fbff7827fa Refactor progress bar & summary line logic
* Centralize the logic about whether to print the progress bar or not in
  the `*_PROGRESS()` macros.
* Centralize the logc about whether to print the summary line or not in
  `FIO_shouldDisplayFileSummary()` and
  `FIO_shouldDisplayMultipleFileSummary()`.
* Make `--progress` work for non-zstd (de)compressors.
* Clean up several edge cases in compression and decompression progress
  printing along the way. E.g. wrong log level, or missing summary line.

One thing I don't like about stdout mode, which sets the display level
to 1, is that warnings aren't displayed. After this PR, we could change
stdout mode from lowering the display level, to defaulting to implied
`--no-progress`. But, I think that deserves a separate PR.
2022-12-14 15:44:09 -08:00
Nick Terrell
e58a39f84e Allow tests to fake stdin/stdout/stderr is a console
We've been unable to effectively test cases where stdin/stdout/stderr
are consoles, because in our test cases they generally aren't. Allow the
command line flags `--fake-std{in,out,err}-is-console` to tell the CLI
to pretend that std{in,out,err} is a console.
2022-12-14 15:44:09 -08:00
Danielle Rozenblit
4dffc35f2e Convert references to https from http 2022-12-14 06:58:35 -08:00
FRex
dc39409a03 Print checksum value for single frame files in cli with -v -l options 2022-12-12 10:53:57 -08:00
Yann Collet
4d82a4d3f2 minor : reorder --help
more logical context grouping
2022-11-22 07:23:45 -08:00
Elliot Gorokhovsky
b1182fe242
Merge pull request #3300 from yiyuaner/dev
Fix the wrong check for buffer overrun in UTIL_mergeFileNamesTable
2022-11-02 15:28:02 -04:00
appleman4321
5334339421 Rewrite checking condition bufStart pointer NULL 2022-11-01 02:31:09 +03:00
yiyuaner
361d86998a Fix the wrong check for buffer overrun in UTIL_mergeFileNamesTable 2022-10-24 20:37:15 +08:00
Yann Collet
9abecfbb7e replaced Zstandard by zstd in man page
identified by @paulmenzel in #3286
2022-10-11 23:32:18 -07:00
Ilya Kurdyukov
2bd70eff06
zstd-pgo: also clean zstd binary from objects
Just a precaution, because it works anyway.
2022-10-06 00:38:23 +07:00
Ilya Kurdyukov
2ffcb2d6a8
fixed zstd-pgo target for GCC
Since your Makefile uses obj/$(HASH_DIR) for object files, this code does not work correctly for GCC. Because profiles are saved in one directory, and are expected in another when reading.

`$(RM) zstd *.o` - this line doesn't delete object files.

Clang stores profiles in the current directory, so the problem doesn't appear when compiling with Clang.

Also this code will work if BUILD_DIR is set.
2022-10-05 17:32:16 +07:00
ctkhanhly
358787764f
Fix for zstd CLI accepts bogus values for numeric parameters (#3268)
* add checks to mal-formed numeric values for memory and memlimit parameters

Signed-off-by: Ly Cao <lycao@fb.com>

* changed errorMsg to a literal string instead of static string in main

* moved bogus numeric error to NEXT_UINT32 + add macro NEXT_TSIZE

Signed-off-by: Ly Cao <lycao@fb.com>

Signed-off-by: Ly Cao <lycao@fb.com>
Co-authored-by: Ly Cao <lycao@fb.com>
2022-09-21 13:20:01 -07:00
Yann Collet
f7251f88b9
Merge pull request #3247 from haampie/fix/grep
Fix make variable
2022-09-19 10:47:38 -07:00