Before this change if we copied files of unknown size, then they lost
their metadata.
This was particularly noticeable using --s3-decompress.
This change adds metadata to Rcat and RcatSized and changes Copy to
pass the metadata in when it calls Rcat for an unknown sized input.
Fixes#6546
Due to Chrome's rather complicated use of file handles when saving
files from the download windows, rclone was attempting to truncate a
closed file.
The file appeared closed due to the handling of 0 length files.
This patch removes the check for the file being closed in the
WriteFileHandle.Truncate call. This is safe because the only action
this method takes is to emit an error message if the file is the wrong
size.
See: https://forum.rclone.org/t/google-drive-cannot-save-files-directly-from-browser-to-gdrive-mounted-path/17992/
This allows reads to only read part of the file and it keeps on disk a
cache of what parts of each file have been loaded.
File data itself is kept in sparse files.
--vfs-read-wait duration Time to wait for in-sequence read before seeking. (default 5ms)
--vfs-write-wait duration Time to wait for in-sequence write before giving error. (default 1s)
See: https://forum.rclone.org/t/constantly-high-iowait-add-log/14156
- Change rclone/fs interfaces to accept context.Context
- Update interface implementations to use context.Context
- Change top level usage to propagate context to lover level functions
Context propagation is needed for stopping transfers and passing other
request-scoped values.
This makes WriteAt for non cached files wait a short time if it gets
an out of order write (which would normally cause an error) to see if
the gap will be filled with an in order write.
The makes the SFTP backend work fine with --vfs-cache-mode off
The purpose of this is to make it easier to maintain and eventually to
allow the rclone backends to be re-used in other projects without
having to use the rclone configuration system.
The new code layout is documented in CONTRIBUTING.
If we open a file for r/w without the cache we now always return a
handle and return an error if the file is ever read from. This fixes
incompatibility with cmount under windows.
* Tests for VFS layer
* Small fixes found during testing
* Fix Close, Flush and Release behaviour for ReadFileHandle and WriteFileHandle
* Fix nil object bugs on File
