mirror of
https://github.com/rclone/rclone.git
synced 2025-11-23 21:44:49 +02:00
pool: add --max-buffer-memory to limit total buffer memory usage
This commit is contained in:
@@ -1526,6 +1526,24 @@ of the remote which may be desirable.
|
||||
Setting this to a negative number will make the backlog as large as
|
||||
possible.
|
||||
|
||||
### --max-buffer-memory=SIZE {#max-buffer-memory}
|
||||
|
||||
If set, don't allocate more than SIZE amount of memory as buffers. If
|
||||
not set or set to `0` or `off` this will not limit the amount of memory
|
||||
in use.
|
||||
|
||||
This includes memory used by buffers created by the `--buffer` flag
|
||||
and buffers used by multi-thread transfers.
|
||||
|
||||
Most multi-thread transfers do not take additional memory, but some do
|
||||
depending on the backend (eg the s3 backend for uploads). This means
|
||||
there is a tension between total setting `--transfers` as high as
|
||||
possible and memory use.
|
||||
|
||||
Setting `--max-buffer-memory` allows the buffer memory to be
|
||||
controlled so that it doesn't overwhelm the machine and allows
|
||||
`--transfers` to be set large.
|
||||
|
||||
### --max-delete=N ###
|
||||
|
||||
This tells rclone not to delete more than N files. If that limit is
|
||||
@@ -1785,6 +1803,14 @@ This will work with the `sync`/`copy`/`move` commands and friends
|
||||
mount` and `rclone serve` if `--vfs-cache-mode` is set to `writes` or
|
||||
above.
|
||||
|
||||
Most multi-thread transfers do not take additional memory, but some do
|
||||
(for example uploading to s3). In the worst case memory usage can be
|
||||
at maximum `--transfers` * `--multi-thread-chunk-size` *
|
||||
`--multi-thread-streams` or specifically for the s3 backend
|
||||
`--transfers` * `--s3-chunk-size` * `--s3-concurrency`. However you
|
||||
can use the the [--max-buffer-memory](/docs/#max-buffer-memory) flag
|
||||
to control the maximum memory used here.
|
||||
|
||||
**NB** that this **only** works with supported backends as the
|
||||
destination but will work with any backend as the source.
|
||||
|
||||
|
||||
Reference in New Issue
Block a user