1
0
mirror of https://github.com/rclone/rclone.git synced 2025-01-19 04:47:54 +02:00
Nick Craig-Wood 9b5308144f s3: Reduce memory usage streaming files by reducing max stream upload size
Before this change rclone would allow the user to stream (eg with
rclone mount, rclone rcat or uploading google photos or docs) 5TB
files.  This meant that rclone allocated 4 * 525 MB buffers per
transfer which is way too much memory by default.

This change makes rclone use the configured chunk size for streamed
uploads.  This is 5MB by default which means that rclone can stream
upload files up to 48GB by default staying below the 10,000 chunks
limit.

This can be increased with --s3-chunk-size if necessary.

If rclone detects that a file is being streamed to s3 it will make a
single NOTICE level log stating the limitation.

This fixes the enormous memory usage.

Fixes 
See: https://forum.rclone.org/t/how-much-memory-does-rclone-need/12743
2019-11-09 15:55:19 +00:00
2019-11-02 15:26:01 +00:00
2019-10-29 17:41:56 +00:00
2019-06-19 11:59:46 +01:00
2019-09-17 10:12:32 +01:00
2019-10-29 17:41:56 +00:00
2019-10-29 17:41:56 +00:00
2019-10-20 00:33:16 +03:00
2019-10-26 11:04:54 +01:00
2019-10-26 11:04:54 +01:00
2019-10-26 11:04:54 +01:00
2019-10-26 11:04:54 +01:00
2019-09-29 11:05:10 +01:00
2019-10-08 21:55:12 +01:00
2019-11-02 15:26:01 +00:00
2019-11-02 15:26:01 +00:00

rclone logo

Website | Documentation | Download | Contributing | Changelog | Installation | Forum

Build Status Windows Build Status Build Status CircleCI Go Report Card GoDoc Docker Pulls

Rclone

Rclone ("rsync for cloud storage") is a command line program to sync files and directories to and from different cloud storage providers.

Storage providers

Please see the full list of all storage providers and their features

Features

  • MD5/SHA-1 hashes checked at all times for file integrity
  • Timestamps preserved on files
  • Partial syncs supported on a whole file basis
  • Copy mode to just copy new/changed files
  • Sync (one way) mode to make a directory identical
  • Check mode to check for file hash equality
  • Can sync to and from network, e.g. two different cloud accounts
  • Optional large file chunking (Chunker)
  • Optional encryption (Crypt)
  • Optional cache (Cache)
  • Optional FUSE mount (rclone mount)
  • Multi-threaded downloads to local disk
  • Can serve local or remote files over HTTP/WebDav/FTP/SFTP/dlna

Installation & documentation

Please see the rclone website for:

Downloads

License

This is free software under the terms of MIT the license (check the COPYING file included in this package).

Description
"rsync for cloud storage" - Google Drive, S3, Dropbox, Backblaze B2, One Drive, Swift, Hubic, Wasabi, Google Cloud Storage, Yandex Files
Readme 342 MiB
Languages
Go 98.6%
Shell 0.4%
HTML 0.3%
Python 0.3%
JavaScript 0.2%
Other 0.1%