go/rclone
1
0
Fork 0
mirror of https://github.com/rclone/rclone.git synced 2025-03-17 20:27:52 +02:00
Code Issues Releases Activity

operations: fix hashing problem in integration tests

Browse Source 
Before this change backends which supported more than one hash (eg
pcloud) or backends which wrapped backends supporting more than one
hash (combine) would fail the TestMultithreadCopy and
TestMultithreadCopyAbort with an error like

    Failed to make new multi hasher: requested set 000001ff contains unknown hash types

This was caused by the tests limiting the globally available hashes to
the first hash supplied by the backend.

This was added in this commit

d5d28a7513b09701 operations: fix overwrite of destination when multi-thread transfer fails

to overcome the tests taking >100s on the local backend because they
made every single hash that the local backend. It brought this time
down to 20s.

This commit fixes the problem and retains the CPU speedup by only
applying the fix from the original commit if the destination backend
is the local backend. This fixes the common case (testing on the local
backend). This does not fix the problem for a backend which wraps the
local backend (eg combine) but this is run only on the integration
test machine and not on all the CI.
This commit is contained in:
Nick Craig-Wood 2024-06-12 10:52:58 +01:00
parent a341065b8d
commit d068e0b1a9
1 changed files with 7 additions and 5 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

12
fs/operations/multithread_test.go
View File

@ -121,11 +121,13 @@ func skipIfNotMultithread(ctx context.Context, t *testing.T, r *fstest.Run) int
t.Skip("multithread writing not supported")
}
// Only support one hash otherwise we end up spending a huge amount of CPU on hashing!
oldHashes := hash.SupportOnly([]hash.Type{r.Fremote.Hashes().GetOne()})
t.Cleanup(func() {
_ = hash.SupportOnly(oldHashes)
})
// Only support one hash for the local backend otherwise we end up spending a huge amount of CPU on hashing!
if r.Fremote.Features().IsLocal {
oldHashes := hash.SupportOnly([]hash.Type{r.Fremote.Hashes().GetOne()})
t.Cleanup(func() {
_ = hash.SupportOnly(oldHashes)
})
}
ci := fs.GetConfig(ctx)
chunkSize := int(ci.MultiThreadChunkSize)