1
0
mirror of https://github.com/facebook/zstd.git synced 2025-03-07 09:26:03 +02:00

457 Commits

Author SHA1 Message Date
Yonatan Komornik
cc0657f27d
AsyncIO compression part 2 - added async read and asyncio to compression code (#3022)
* Compression asyncio:
- Added asyncio functionality for compression flow
- Added ReadPool for async reads, implemented in both comp and decomp flows
2022-01-31 15:43:41 -08:00
Nick Terrell
0b70da6277
Merge pull request #3020 from terrelln/cli-tests
Add new CLI testing platform
2022-01-31 10:02:27 -08:00
Yann Collet
c9072dd60a
Merge pull request #3027 from brailovich/dev
fix for -r on empty directory
2022-01-29 15:40:43 -08:00
Nick Terrell
495dcb839a [zstdcli] Fix option detection for --auto-threads
The option `--auto-threads` should still be accepted and parsed, even if
`ZSTD_MULTITHREAD` is not defined. It doesn't mean anything, but we
should still accept the option. Since we want scripts to be able to work
generically.

This bug was caught by tests I added to the new testing framework.
2022-01-27 13:56:59 -08:00
Nick Terrell
e60eba58bf Print zlib/lz4/lzma library versions in verbose version output
Knowing the version of zlib/lz4/lzma we're linking against is very
useful for debugging issues with those libraries, so print it out in the
verbosity 4 version output.

Also print this information at the top of `playTests.sh`.
2022-01-26 18:25:58 -08:00
brailovich
beb4872241
Update zstdcli.c 2022-01-26 16:51:18 -08:00
brailovich
4021b78437
fix for error message in recursive mode for an empty folder
-r on empty directory resulted in zstd waiting input from stdin. now zstd exits without error and prints a warning message explaining why no processing happened (no files or directories to process).
2022-01-24 17:42:21 -08:00
Yonatan Komornik
1598e6c634
Async write for decompression (#2975)
* Async IO decompression:
- Added --[no-]asyncio flag for CLI decompression.
- Replaced dstBuffer in decompression with a pool of write jobs.
- Added an ability to execute write jobs in a separate thread.
- Added an ability to wait (join) on all jobs in a thread pool (queued and running).
2022-01-21 13:55:41 -08:00
Elliot Gorokhovsky
71c0c07c19 Allow user to specify memory limit for dictionary training 2021-12-14 14:29:01 -05:00
binhdvo
38dfc4699e
Imply -q when stderr is not a tty (#2884)
* Imply -q when stderr is not a tty
2021-12-07 16:56:19 -05:00
binhdvo
7abebc847b
Clarify documentation for -c (#2883) 2021-11-29 14:10:43 -05:00
Kevin Svetlitski
365c91194c Ensure print*CParams functions are only defined when used 2021-11-11 12:14:56 -08:00
Kevin Svetlitski
63fe6198ed Display --zstd= subparameters in command-line ready form in verbose mode 2021-11-10 17:20:43 -08:00
Kevin Svetlitski
0665d4c1c2 Display command line parameters with concrete values in verbose mode 2021-11-05 12:01:20 -07:00
stanjo74
52598d54e9
Limit train samples (#2809)
* Limit training samples size to 2GB

* simplified DISPLAYLEVEL() macro to use global vqriable instead of local.

* refactored training samples loading

* fixed compiler warning

* addressed comments from the pull request

* addressed @terrelln comments

* missed some fixes

* fixed type mismatch

* Fixed bug passing estimated number of samples rather insted of the loaded number of samples.
Changed unit conversion not to use bit-shifts.

* fixed a declaration after code

* fixed type conversion compile errors

* fixed more type castting

* fixed more type mismatching

* changed sizes type to size_t

* move type casting

* more type cast fixes
2021-10-04 17:47:52 -07:00
senhuang42
b5c35d7ea3 Use new paramSwitch enum for LCM, row matchfinder, and block splitter 2021-09-21 14:22:02 -04:00
makise-homura
a5f518ae27 Change zstdcli's main() declaration due to -Wmain on some compilers 2021-07-14 19:55:47 +03:00
makise-homura
d4ad02c721 Add support for MCST LCC compiler 2021-07-10 03:57:06 +03:00
Binh Vo
6a46e38deb Add option to use logical cores for default threads 2021-06-16 15:46:17 -04:00
binhdvo
89127e5ee2
Merge pull request #2705 from binhdvo/bootcamp
Add support for negative values in advanced flags
2021-06-11 14:08:23 -04:00
Binh Vo
6fad35c6a1 Add support for negative levels in --adapt=min and --adapt=max" 2021-06-11 12:13:09 -04:00
W. Felix Handte
464bfb022e In Verbose Mode, Preserve Full Precision Where Possible 2021-06-10 12:53:07 -04:00
binhdvo
325952f878
Revert "Add support for --long-param flag, fix #2104" 2021-06-09 15:35:43 -04:00
Binh Vo
6583fa3f0a Add support for --long-param flag 2021-06-09 14:07:52 -04:00
Binh Vo
1e17184ad0 Add documentation for --patch-from 2021-06-03 11:12:27 -04:00
sen
6030cdfede
Add --progress flag (#2595) 2021-05-06 14:50:28 -04:00
W. Felix Handte
33f3e293e8 Allow Reading from Block Devices with --force 2021-05-04 16:25:26 -04:00
Nick Terrell
4694423c4f Add and integrate lazy row hash strategy 2021-04-07 09:53:34 -07:00
Nick Terrell
a494308ae9 [copyright][license] Switch to yearless copyright and some cleanup in the linux-kernel files
* Switch to yearless copyright per FB policy
* Fix up SPDX-License-Identifier lines in `contrib/linux-kernel` sources
* Add zstd copyright/license header to the `contrib/linux-kernel` sources
* Update the `tests/test-license.py` to check for yearless copyright
* Improvements to `tests/test-license.py`
* Check `contrib/linux-kernel` in `tests/test-license.py`
2021-03-30 10:30:43 -07:00
senhuang42
444c4650a0 Add newline to end of cli help message 2021-02-17 12:30:42 -05:00
Nick Terrell
54a4998a80 Add basic tracing functionality 2021-02-05 16:28:52 -08:00
W. Felix Handte
8b6a4b5b7c Allow Input From Console When --force is Passed
Also update option flag documentation.
2021-01-11 17:53:20 -05:00
Yann Collet
3324e87cff Added library version check 2021-01-07 10:37:27 -08:00
Nick Terrell
66e811d782 [license] Update year to 2021 2021-01-04 17:53:52 -05:00
senhuang42
7259b258d1 Add callsites to zstdcli.c and tests to playTests.sh 2020-10-07 13:47:38 -04:00
Yann Collet
f7d4943788
Merge pull request #2330 from senhuang42/fix_stdinout_error_messages
Improve error messages on console input/output
2020-10-06 15:58:37 -07:00
senhuang42
ce56810a32 Modify error messages on console input/output 2020-09-28 12:15:18 -04:00
senhuang42
432186cbea Add FIO_determineHasStdinInput() function and member to fCtx 2020-09-24 15:55:30 -04:00
Yann Collet
05622992d5
Merge pull request #2292 from senhuang42/multifile_status_update_fio_refactor
Make multifile (de)compression print out a summary
2020-09-14 14:06:26 -07:00
Yann Collet
e583e0be8c
Merge pull request #2299 from senhuang42/env_var_num_threads
Allow environment variable to specify number of threads for compression
2020-09-14 14:04:19 -07:00
senhuang42
a71963c7b8 nbThreads instead of numThreads 2020-09-09 12:40:00 -04:00
senhuang42
243c8dfb41 Add include guards for init_numThreads() 2020-09-08 09:26:16 -04:00
senhuang42
136a0673f6 Add quick documentation for ZSTD_NUMTHREADS in the code 2020-09-07 18:19:31 -04:00
senhuang42
888c385a49 Change param name to ZSTDCLI_NUMTHREADS_DEFAULT 2020-09-07 18:11:42 -04:00
senhuang42
972e063219 Change default num_threads value 2020-09-07 18:09:48 -04:00
senhuang42
5123496104 Adjust function signatures 2020-09-07 13:13:05 -04:00
senhuang42
48bca10792 Address memory leak in CLI from fCtx 2020-09-03 10:14:04 -04:00
senhuang42
3a7d625d6b Cleanup comments, add function to set FIO_ctx_t.nbFilesProcessed 2020-09-01 12:54:21 -04:00
senhuang42
a6414f1247 Integrate refactor into status print for multifiles, adjust logic for printing as needed 2020-09-01 12:34:43 -04:00
senhuang42
565f116a56 Change name to ZSTD_NUMTHREADS 2020-09-01 09:02:23 -04:00