krak/zstd
krak/zstd
1
0
Fork 0
mirror of https://github.com/facebook/zstd.git synced 2025-03-07 09:26:03 +02:00
Code Issues Releases 1 Activity
10,928 Commits 25 Branches 76 Tags
Commit Graph

182 Commits

Author SHA1 Message Date
Felix Handte
490163ac73
Merge pull request #3963 from felixhandte/also-handle-hidden-files-output-dir-mirror 
Remove Erroneous Exclusion of Hidden Files and Folders in `--output-dir-mirror`
 2024-03-14 10:21:00 -04:00
W. Felix Handte
86b8e39a84 Remove Erroneous Exclusion of Hidden Files and Folders in --output-dir-mirror 2024-03-13 16:33:30 -04:00
W. Felix Handte
d6ee2d5d24 Use utimensat() on FreeBSD 
FreeBSD only claims to support POSIX 2001 [0]. But they do in fact support
`utimensat()`. This adds a specific check to opt them in to using it. This
value was selected by consulting [1].

See discussion on #3952.

Further addresses #3748.

[0] 937a005585/sys/sys/unistd.h (L96)

[1] https://docs.freebsd.org/en/books/porters-handbook/versions/
 2024-03-13 13:26:21 -07:00
Christoph Grüninger
b921f1aad6 Reduce scope of variables 
This improves readability, keeps variables local, and
prevents the unintended use (e.g. typo) later on.
Found by Cppcheck (variableScope)
 2024-02-11 22:00:03 +01:00
W. Felix Handte
f746c37d00 Use File Descriptor in Setting Stat on Output File 
Note that the `fd` is only valid while the file is still open. So we need to
move the setting calls to before we close the file. However! We cannot do so
with the `utime()` call (even though `futimens()` exists) because the follow-
ing `close()` call to the `fd` will reset the atime of the file. So it seems
the `utime()` call has to happen after the file is closed.
 2023-02-06 14:02:31 -08:00
W. Felix Handte
a5a2418df4 Introduce Variants of Some Functions that Take Optional File Descriptors 
Somewhat surprisingly, calling `fchmod()` is non-trivially faster than calling
`chmod()`, and so on.

This commit introduces alternate variants to some common file util functions
that take an optional fd. If present, they call the `f`-variant of the
underlying function. Otherwise, they fall back to the regular filename-taking
version of the function.
 2023-02-06 13:55:34 -08:00
Yann Collet
8c85b29e32 disable --rm on -o command 
make it more similar to -c (aka `stdout`) convention.
 2023-01-25 16:09:25 -08:00
W. Felix Handte
7a8c8f3fe7 Easy: Print Mode as Octal in chmod() Trace 2023-01-18 11:57:54 -08:00
W. Felix Handte
0d2d460223 Mimic gzip chown(gid), chmod(), chown(uid) Behavior 
Avoids a race condition in which we unintentionally open up permissions to
the wrong group.
 2023-01-18 11:57:54 -08:00
W. Felix Handte
0382076af7 Re-Use stat_t in FIO_compressFilename_srcFile() 2023-01-18 11:33:07 -08:00
Nick Terrell
b6e8112261 [util] Add traces enabled by --trace-file-stat 
Print traces to stderr when --trace-file-stat is passed.
We trace all functions that read and write file metadata.
 2022-12-21 17:19:34 -08:00
W. Felix Handte
5d693cc38c Coalesce Almost All Copyright Notices to Standard Phrasing 
```
for f in $(find . \( -path ./.git -o -path ./tests/fuzz/corpora -o -path ./tests/regression/data-cache -o -path ./tests/regression/cache \) -prune -o -type f); do sed -i '/Copyright .* \(Yann Collet\)\|\(Meta Platforms\)/ s/Copyright .*/Copyright (c) Meta Platforms, Inc. and affiliates./' $f; done

git checkout HEAD -- build/VS2010/libzstd-dll/libzstd-dll.rc build/VS2010/zstd/zstd.rc tests/test-license.py contrib/linux-kernel/test/include/linux/xxhash.h examples/streaming_compression_thread_pool.c lib/legacy/zstd_v0*.c lib/legacy/zstd_v0*.h
nano ./programs/windres/zstd.rc
nano ./build/VS2010/zstd/zstd.rc
nano ./build/VS2010/libzstd-dll/libzstd-dll.rc
```
 2022-12-20 12:52:34 -05:00
W. Felix Handte
8927f985ff Update Copyright Headers 'Facebook' -> 'Meta Platforms' 
```
for f in $(find . \( -path ./.git -o -path ./tests/fuzz/corpora \) -prune -o -type f);
do
  sed -i 's/Facebook, Inc\./Meta Platforms, Inc. and affiliates./' $f;
done
```
 2022-12-20 12:37:57 -05:00
Nick Terrell
e58a39f84e Allow tests to fake stdin/stdout/stderr is a console 
We've been unable to effectively test cases where stdin/stdout/stderr
are consoles, because in our test cases they generally aren't. Allow the
command line flags `--fake-std{in,out,err}-is-console` to tell the CLI
to pretend that std{in,out,err} is a console.
 2022-12-14 15:44:09 -08:00
Elliot Gorokhovsky
b1182fe242
Merge pull request #3300 from yiyuaner/dev 
Fix the wrong check for buffer overrun in UTIL_mergeFileNamesTable
 2022-11-02 15:28:02 -04:00
appleman4321
5334339421 Rewrite checking condition bufStart pointer NULL 2022-11-01 02:31:09 +03:00
yiyuaner
361d86998a Fix the wrong check for buffer overrun in UTIL_mergeFileNamesTable 2022-10-24 20:37:15 +08:00
Elliot Gorokhovsky
28ceb63503
Merge pull request #3220 from embg/issue3200 
Disallow empty string as argument for --output-dir-flat and --output-dir-mirror
 2022-08-01 14:04:57 -04:00
Yonatan Komornik
ae4670466c
stdin multiple file fixes (#3222) 
* Fixes for https://github.com/facebook/zstd/issues/3206 - bugs when handling stdin as part of multiple files.

* new line at end of multiple-files.sh
 2022-07-29 16:13:07 -07:00
Qiongsi Wu
b1bbb0eb4c
[AIX] Fix Compiler Flags and Bugs on AIX to Pass All Tests (#3219) 
* Fixing compiler warnings

* Replace the old -s flag with the -Wl,-s flag

* Fixing compiler warnings

* Fixing the linker strip flag and tests/code not working as expected on AIX
 2022-07-29 12:21:59 -07:00
Elliot Gorokhovsky
e1873ad576 Fix buffer underflow for null dir1 2022-07-29 11:10:47 -07:00
Dominique Pelle
b772f53952 Typo and grammar fixes 2022-03-12 08:58:04 +01:00
Xi Ruoyao
d109cef201
fix the assertion in readLinesFromFile (#3084) 
* fix the assertion in readLinesFromFile

When the file is not terminated by endline, readLineFromFile will append
a '\0' for the last line.  In this case pos + lineLength == dstCapacity.

* test: don't print very long text garbage
 2022-03-04 14:56:44 -05:00
binhdvo
d7e1736375
Fix build for cygwin/bsd (#2882) 2021-11-29 14:11:39 -05:00
Dimitris Apostolou
ebbd675998
Fix typos 2021-11-13 10:04:04 +02:00
Yann Collet
eab692211e removed pretty-print of sizes in benchmark 
This is less appropriate for this mode :
benchmark is about accuracy,
it's important to read the exact values.
 2021-09-03 12:51:02 -07:00
W. Felix Handte
a719edbbc3 Pull utime() Call into Helper 2021-08-04 14:49:00 -04:00
Binh Vo
6a46e38deb Add option to use logical cores for default threads 2021-06-16 15:46:17 -04:00
W. Felix Handte
2af3687c50 Switch to Binary Size Prefixes (e.g., "MB" -> "MiB") 
Suggested by @aqrit, a little more verbose, but hopefully addresses a real
ambiguity.
 2021-06-10 12:53:07 -04:00
W. Felix Handte
9c340ce014 Require -vv to Enable Full Precision 2021-06-10 12:53:07 -04:00
W. Felix Handte
93bb368d74 Change Suffix (e.g., "G" -> " GB") 2021-06-10 12:53:07 -04:00
W. Felix Handte
464bfb022e In Verbose Mode, Preserve Full Precision Where Possible 2021-06-10 12:53:07 -04:00
W. Felix Handte
9b67219b1e Fix Integer Constants; Fix Comparison 2021-06-10 12:53:07 -04:00
W. Felix Handte
bbb81c8801 Avoid snprintf() in Preparing Human-Readable Sizes; Improve Formatting 
This produces the following formatting:

   Size    | `zstd` | `ls -lh`
---------- | ------ | --------
1          | 1      | 1
12         | 12     | 12
123        | 123    | 123
1234       | 1.21K  | 1.3K
12345      | 12.1K  | 13K
123456     | 121K   | 121K
1234567    | 1.18M  | 1.2M
12345678   | 11.8M  | 12M
123456789  | 118M   | 118M
1234567890 | 1.15G  | 1.2G
999        | 999    | 999
1000       | 1000   | 1000
1001       | 1001   | 1001
1023       | 1023   | 1023
1024       | 1.000K | 1.0K
1025       | 1.00K  | 1.1K
999999     | 977K   | 977K
1000000    | 977K   | 977K
1000001    | 977K   | 977K
1023999    | 1000K  | 1000K
1024000    | 1000K  | 1000K
1024001    | 1000K  | 1001K
1048575    | 1024K  | 1.0M
1048576    | 1.000M | 1.0M
1048577    | 1.00M  | 1.1M

This was produced with the following invocation:

```
for N in 1 12 123 1234 12345 123456 1234567 12345678 123456789 1234567890 999 1000 1001 1023 1024 1025 999999 1000000 1000001 1023999 1024000 1024001 1048575 1048576 1048577; do
  head -c $N /dev/urandom > r$N
done
./zstd -i1 -b1 -S r1 r12 r123 r1234 r12345 r123456 r1234567 r12345678 r123456789 r1234567890 r999 r1000 r1001 r1023 r1024 r1025 r999999 r1000000 r1000001 r1023999 r1024000 r1024001 r1048575 r1048576 r1048577
```
 2021-06-10 12:53:07 -04:00
Scott Baker
8e0a9695d7 Attempt to fix a failing test with help from @aqrit 2021-06-10 12:53:07 -04:00
Scott Baker
1eb852854b Some fixes to address things @felixhandte found 2021-06-10 12:53:07 -04:00
Scott Baker
376a2730a8 Try enabling the BIG strings now the unsigned long long is in effect 2021-06-10 12:53:07 -04:00
Scott Baker
20b9b00b41 Try unsigned long long 2021-06-10 12:53:07 -04:00
Scott Baker
64385ef7cb Update humanSize() to skip the big numbers (it requires 64 bit) 2021-06-10 12:53:07 -04:00
Scott Baker
1ef6f3d079 Use unsigned long instead to help with some tests 2021-06-10 12:53:07 -04:00
Scott Baker
e5fc830795 human_size() should use size_t 2021-06-10 12:53:07 -04:00
Scott Baker
eefdbcd93a Make the variable types match 2021-06-10 12:53:07 -04:00
Scott Baker
b6b23dfe64 Convert names to CamelCase 2021-06-10 12:53:07 -04:00
Scott Baker
b70175e5ec Put the human_size() function in util.c 2021-06-10 12:53:07 -04:00
Yann Collet
df05b2ba7c fix --filelist compatibility with Windows cr+lf line ending 2021-05-05 18:01:55 -07:00
W. Felix Handte
b87f97b3ea Create Files with Desired Permissions; Avoid chmod(); Remove UTIL_chmod() 2021-05-05 13:10:34 -04:00
W. Felix Handte
33f3e293e8 Allow Reading from Block Devices with --force 2021-05-04 16:25:26 -04:00
Nick Terrell
a494308ae9 [copyright][license] Switch to yearless copyright and some cleanup in the linux-kernel files 
* Switch to yearless copyright per FB policy
* Fix up SPDX-License-Identifier lines in `contrib/linux-kernel` sources
* Add zstd copyright/license header to the `contrib/linux-kernel` sources
* Update the `tests/test-license.py` to check for yearless copyright
* Improvements to `tests/test-license.py`
* Check `contrib/linux-kernel` in `tests/test-license.py`
 2021-03-30 10:30:43 -07:00
Sen Huang
f27e326456 Restrict dictmode regression tests only to advanced API, fix some compiler warnings 2021-03-25 10:39:08 -07:00
Paul Bone
4d6c78fb89 Only set numPhysicalCores if ratio is valid 2021-03-03 10:59:00 +11:00