1
0
mirror of https://github.com/facebook/zstd.git synced 2025-07-06 15:45:37 +02:00
Commit Graph

187 Commits

Author SHA1 Message Date
a5a2418df4 Introduce Variants of Some Functions that Take Optional File Descriptors
Somewhat surprisingly, calling `fchmod()` is non-trivially faster than calling
`chmod()`, and so on.

This commit introduces alternate variants to some common file util functions
that take an optional fd. If present, they call the `f`-variant of the
underlying function. Otherwise, they fall back to the regular filename-taking
version of the function.
2023-02-06 13:55:34 -08:00
0382076af7 Re-Use stat_t in FIO_compressFilename_srcFile() 2023-01-18 11:33:07 -08:00
90597d78ea Merge pull request #3394 from terrelln/issue-3010
[cli-tests] Test file stat read/write
2022-12-27 16:20:05 -08:00
40a7188130 Fix make clangbuild & add CI
Fix the errors for:
* `-Wdocumentation`
* `-Wconversion` except `-Wsign-conversion`
2022-12-21 17:31:04 -08:00
b6e8112261 [util] Add traces enabled by --trace-file-stat
Print traces to stderr when --trace-file-stat is passed.
We trace all functions that read and write file metadata.
2022-12-21 17:19:34 -08:00
5d693cc38c Coalesce Almost All Copyright Notices to Standard Phrasing
```
for f in $(find . \( -path ./.git -o -path ./tests/fuzz/corpora -o -path ./tests/regression/data-cache -o -path ./tests/regression/cache \) -prune -o -type f); do sed -i '/Copyright .* \(Yann Collet\)\|\(Meta Platforms\)/ s/Copyright .*/Copyright (c) Meta Platforms, Inc. and affiliates./' $f; done

git checkout HEAD -- build/VS2010/libzstd-dll/libzstd-dll.rc build/VS2010/zstd/zstd.rc tests/test-license.py contrib/linux-kernel/test/include/linux/xxhash.h examples/streaming_compression_thread_pool.c lib/legacy/zstd_v0*.c lib/legacy/zstd_v0*.h
nano ./programs/windres/zstd.rc
nano ./build/VS2010/zstd/zstd.rc
nano ./build/VS2010/libzstd-dll/libzstd-dll.rc
```
2022-12-20 12:52:34 -05:00
8927f985ff Update Copyright Headers 'Facebook' -> 'Meta Platforms'
```
for f in $(find . \( -path ./.git -o -path ./tests/fuzz/corpora \) -prune -o -type f);
do
  sed -i 's/Facebook, Inc\./Meta Platforms, Inc. and affiliates./' $f;
done
```
2022-12-20 12:37:57 -05:00
e58a39f84e Allow tests to fake stdin/stdout/stderr is a console
We've been unable to effectively test cases where stdin/stdout/stderr
are consoles, because in our test cases they generally aren't. Allow the
command line flags `--fake-std{in,out,err}-is-console` to tell the CLI
to pretend that std{in,out,err} is a console.
2022-12-14 15:44:09 -08:00
ae4670466c stdin multiple file fixes (#3222)
* Fixes for https://github.com/facebook/zstd/issues/3206 - bugs when handling stdin as part of multiple files.

* new line at end of multiple-files.sh
2022-07-29 16:13:07 -07:00
ebbd675998 Fix typos 2021-11-13 10:04:04 +02:00
eab692211e removed pretty-print of sizes in benchmark
This is less appropriate for this mode :
benchmark is about accuracy,
it's important to read the exact values.
2021-09-03 12:51:02 -07:00
a719edbbc3 Pull utime() Call into Helper 2021-08-04 14:49:00 -04:00
6a46e38deb Add option to use logical cores for default threads 2021-06-16 15:46:17 -04:00
464bfb022e In Verbose Mode, Preserve Full Precision Where Possible 2021-06-10 12:53:07 -04:00
bbb81c8801 Avoid snprintf() in Preparing Human-Readable Sizes; Improve Formatting
This produces the following formatting:

   Size    | `zstd` | `ls -lh`
---------- | ------ | --------
1          | 1      | 1
12         | 12     | 12
123        | 123    | 123
1234       | 1.21K  | 1.3K
12345      | 12.1K  | 13K
123456     | 121K   | 121K
1234567    | 1.18M  | 1.2M
12345678   | 11.8M  | 12M
123456789  | 118M   | 118M
1234567890 | 1.15G  | 1.2G
999        | 999    | 999
1000       | 1000   | 1000
1001       | 1001   | 1001
1023       | 1023   | 1023
1024       | 1.000K | 1.0K
1025       | 1.00K  | 1.1K
999999     | 977K   | 977K
1000000    | 977K   | 977K
1000001    | 977K   | 977K
1023999    | 1000K  | 1000K
1024000    | 1000K  | 1000K
1024001    | 1000K  | 1001K
1048575    | 1024K  | 1.0M
1048576    | 1.000M | 1.0M
1048577    | 1.00M  | 1.1M

This was produced with the following invocation:

```
for N in 1 12 123 1234 12345 123456 1234567 12345678 123456789 1234567890 999 1000 1001 1023 1024 1025 999999 1000000 1000001 1023999 1024000 1024001 1048575 1048576 1048577; do
  head -c $N /dev/urandom > r$N
done
./zstd -i1 -b1 -S r1 r12 r123 r1234 r12345 r123456 r1234567 r12345678 r123456789 r1234567890 r999 r1000 r1001 r1023 r1024 r1025 r999999 r1000000 r1000001 r1023999 r1024000 r1024001 r1048575 r1048576 r1048577
```
2021-06-10 12:53:07 -04:00
1eb852854b Some fixes to address things @felixhandte found 2021-06-10 12:53:07 -04:00
20b9b00b41 Try unsigned long long 2021-06-10 12:53:07 -04:00
1ef6f3d079 Use unsigned long instead to help with some tests 2021-06-10 12:53:07 -04:00
e5fc830795 human_size() should use size_t 2021-06-10 12:53:07 -04:00
b6b23dfe64 Convert names to CamelCase 2021-06-10 12:53:07 -04:00
b70175e5ec Put the human_size() function in util.c 2021-06-10 12:53:07 -04:00
b87f97b3ea Create Files with Desired Permissions; Avoid chmod(); Remove UTIL_chmod() 2021-05-05 13:10:34 -04:00
33f3e293e8 Allow Reading from Block Devices with --force 2021-05-04 16:25:26 -04:00
a494308ae9 [copyright][license] Switch to yearless copyright and some cleanup in the linux-kernel files
* Switch to yearless copyright per FB policy
* Fix up SPDX-License-Identifier lines in `contrib/linux-kernel` sources
* Add zstd copyright/license header to the `contrib/linux-kernel` sources
* Update the `tests/test-license.py` to check for yearless copyright
* Improvements to `tests/test-license.py`
* Check `contrib/linux-kernel` in `tests/test-license.py`
2021-03-30 10:30:43 -07:00
f27e326456 Restrict dictmode regression tests only to advanced API, fix some compiler warnings 2021-03-25 10:39:08 -07:00
a774c57973 Use umask() to Constrain Created File Permissions
This commit addresses #2491.

Note that a downside of this solution is that it is global: `umask()` affects
all file creation calls in the process. I believe this is safe since
`fileio.c` functions should only ever be used in the zstd binary, and these
are (almost) the only files ever created by zstd, and AIUI they're only
created in a single thread. So we can get away with messing with global state.

Note that this doesn't change the permissions of files created by `dibio.c`.
I'm not sure what those should be...
2021-02-17 15:27:39 -05:00
66e811d782 [license] Update year to 2021 2021-01-04 17:53:52 -05:00
93d63eaeb8 Expand UTIL_requireUserConfirmation to include stdin input check 2020-09-24 15:58:06 -04:00
ef11aadc0c Fix UTIL_requireUserConfirmation() declaration 2020-08-26 17:20:37 -04:00
7991c55181 Move logic into new function FIO_removeMultiFilesWarning, add support for decompression 2020-08-26 16:50:20 -04:00
aab11ce3db Unified warning prompts into new function UTIL_requireUserConfirmationToProceed() 2020-08-25 11:25:49 -04:00
51ac0207af Remove UTIL_statFile() and UTIL_statDir(); Decompose Former Call-Sites 2020-08-10 15:28:02 -04:00
93dda988c8 Remove Unused Function UTIL_fileExist() 2020-08-10 15:22:53 -04:00
76878697a4 Re-Organize and Document Prototypes in util.h 2020-08-10 15:16:14 -04:00
0a8aacb4db Use stat() to Check that File Should be chmod()-ed
Rather than special-casing a check for `/dev/null`, this uses `stat()` to
avoid `chmod()`-ing any non-regular file. I believe this is the desirable
behavior. `UTIL_chmod()` is never called on directories at the moment, only
output files.
2020-08-05 12:00:12 -04:00
44fa052599 Introduce Variants of Various UTIL Functions that Take Pre-Populated stat_t Structs
Instead of calling `stat()`, these functions accept the result of a previous
`stat()` call on the file in question, which will allow us to make multiple
decisions around a file without redundant `stat()` calls.
2020-08-05 01:00:06 -04:00
b6e24bc4dc Rename UTIL_getFileStat() -> UTIL_statFile() and UTIL_getDirectoryStat() -> UTIL_statDir()
I want to introduce versions of many of these functions that take pre-
populated `stat_t` objects and use those rather than doing their own redundant
`stat()` internally. These functions will have `...Stat()` suffixes. So this
commit renames these existing functions into the active voice, to avoid
confusion.
2020-08-05 00:40:16 -04:00
1a1003f996 Mark stat_t Arg to UTIL_setFileStat() const 2020-08-05 00:35:21 -04:00
b11bea56a5 Introduce Dedicated Helper to Call stat() 2020-08-05 00:21:21 -04:00
9a8ccd4ba3 Add output-dir-mirror option 2020-06-24 22:12:11 -07:00
7dcca6bc64 Also Move programs/ Directory to Relative Includes 2020-05-04 15:20:26 -04:00
ac58c8d720 Fix copyright and license lines
* All copyright lines now have -2020 instead of -present
* All copyright lines include "Facebook, Inc"
* All licenses are now standardized

The copyright in `threading.{h,c}` is not changed because it comes from
zstdmt.

The copyright and license of `divsufsort.{h,c}` is not changed.
2020-03-26 17:02:06 -07:00
96ee20758c assembleFNT() can no longer fail 2019-11-26 15:44:33 -08:00
aaab618ae9 pushed aside stdio.h too
since only UTIL_DISPLAY() depended on it.
2019-11-26 15:25:32 -08:00
7543cd055c moved UTIL_DISPLAY() inside util.c 2019-11-26 15:21:58 -08:00
a684b82774 util: isolated some dependencies
from *.h to *.c
so that they don't get transitively included
into users of util.h.
2019-11-26 15:16:53 -08:00
2d9fad4f52 fixed minor VS warning, on parameter difference
complaining about a `const` property on one side but not the other.
2019-11-26 14:53:37 -08:00
f622c0adf3 switched UTIL_refFilename() to an assert() 2019-11-26 14:48:23 -08:00
c71bd45a3b Merge branch 'dev' into ahmed_file 2019-11-26 11:20:26 -08:00
9a3de0a535 changed name from createX to assembleX
shows that the resulting object just takes ownership of provided buffer.
2019-11-25 15:34:55 -08:00