* Adds "aix/ppc64" to the cross-compile target list.
* Including AIX in the build tag of "metadata_other.go".
* Excluding AIX from the main ncdu build tags.
* Marking AIX as an unsupported platform for ncdu.
* Excluding AIX from the fallback redirect implementation.
* Excluding AIX from unix build tags to avoid undefined unix.WNOHANG.
We were seeing a (non-fatal) error in our logs:
```
Failed to seek log file to end: seek /proc/1/fd/1: illegal seek
```
Because we open the log file with O_APPEND,
we don't need to manually seek to the end.
As https://pkg.go.dev/os#File.Seek also confirms
that the behavior of `Seek` is not specified
if the file has been opened with O_APPEND,
remove the `Seek` call.
Before this change, you had to modify a fragile data-structure
containing all providers. This often led to things being out of order,
duplicates and conflicts whilst merging. As well as the changes for
one provider being in different places across the file.
After this change, new providers are defined in an easy to edit YAML file,
one per provider.
The config output has been tested before and after for all providers
and any changes are cosmetic only.
Before this change --no-traverse was calling NewObject on directories
(where it would always fail) as well as files. This was very
noticeable when doing syncs with --max-age which were only
transferring a small number of objects. This should have been very
quick, but the NewObject calls for each directory slowed the sync down
a lot.
This changes replaces the check to see if the source entry is an
Object that got missed out from this commit:
88e30eecbf march: fix deadlock when using --no-traverse - fixes#8656
Before this, String() quoted every part of the config map even if it
wasn't necessary.
The new Human() method removes the quoting and adds the special case
for "true" values.
Before this change bisync adjusted the global MaxCompletedTransfers
variable which caused races.
This adds a SetMaxCompletedTransfers method and uses it in bisync.
The MaxCompletedTransfers global becomes the default. This can be
changed externally if rclone is in use as a library, and the commit
history indicates that MaxCompletedTransfers was added for exactly
this purpose so we try not to break it here.
Before this change bisync was adjusting MaxCompletedTransfers in order
to clear the done transfers from the stats.
This wasn't working (because it was only clearing one transfer) and
was part of a race adjusting MaxCompletedTransfers.
This fixes the problem by introducing a new method RemoveDoneTransfers
to clear the done transfers explicitly and calling it in bisync.
Before this change, it was possible to have a deadlock when using
--fast-list for a sync if both the source and destination supported
ListR.
This fixes the problem by shortening the locking window.
In this commit:
c63f1865f3 operations: copy: generate stable partial suffix
We made the partial suffix for non inplace copies stable. This was a
hash based off the file fingerprint.
However, given a directory of files which have the same fingerprint
the partial suffix collides. On some backends (eg the local backend)
the fingerprint is just the size and modification time so files with
different contents can collide.
The effect of collisions was hash failures on copy when using
--transfers > 1. These copies invariably retried successfully which
probably explains why this bug hasn't been reported.
This fixes the problem by adding the file name to the hash.
It also makes sure the hash is always represented as 8 hex bytes for
consistency.
Before this change we were reading input from stdin using the terminal
in the default line mode which has a limit of 4095 characters.
The typical culprit was onedrive tokens (which are very long) giving the error
Couldn't decode response: invalid character 'e' looking for beginning of value
This change swaps over to use the github.com/peterh/liner read line
library which does not have that limitation and also enables more
sensible cursor editing.
Fixes#8688#8323#5835
Before this change we used the current context to start the average
loop. This means that if the context came from the rc the average loop
would be cancelled at the end of the rc request leading the speed not
being measured.
This uses the background context for the accounting loop so it doesn't
get cancelled when its parent gets cancelled.
Before this change we opened the connection before allocating memory.
This meant a long wait sometimes for memory and too many connections
open.
Now we allocate the memory first before opening the connection.
Because multipart transfers can need more than one buffer to complete,
if transfers was set very high, it was possible for lots of multipart
transfers to start, grab fewer buffers than chunk size, then deadlock
because no more memory was available.
This fixes the problem by introducing a reservation system which the
multipart transfer uses to ensure it can reserve all the memory for
one chunk before starting.
Before this change rclone about was failing with
cannot unmarshal number 1.0e+18 into Go struct field User.space_amount of type int64
As Box increased Enterprise accounts user.space_amount from 30PB to
1e+18 or 888.178PB returning it as a floating point number, not an integer.
This fix reads it as a float64 and clips it to the maximum value of an
int64 if necessary.
This allows backend config to contain
- `override.var` - set var during remote creation only
- `global.var` - set var in the global config permanently
Fixes#8563
There are some backends (like PikPak) that advertise a precision of
fs.ModTimeNotSupported but do actually return a modtime when asked. In the case
of PikPak, it is because the modtime can be read but not written, and is not
considered reliable enough to use for syncing.
Before this change, testLoggerVsLsf got confused in this scenario (expected a
blank modtime but got non-blank). Adding to the confusion, it only reaches this
code if the backend happens to support md5 hashes, and the fsrc and fdst have
the same precision.
This change fixes the issue by setting the modtime string on both sides to
"none" in this scenario. Note that we can't use "" (blank) because
(operations.ListFormat).AddModTime would replace that with "2006-01-02 15:04:05".
Before this change, two tests (TestServerSideCopyOverSelf and
TestServerSideMoveOverSelf) were checking the wrong Fs in the call to
testLoggerVsLsf. This fixes it by making sure we are testing the same two Fs's
we synced.
