1
0
mirror of https://github.com/rclone/rclone.git synced 2025-11-23 21:44:49 +02:00

pool: fix deadlock with --max-memory and multipart transfers

Because multipart transfers can need more than one buffer to complete,
if transfers was set very high, it was possible for lots of multipart
transfers to start, grab fewer buffers than chunk size, then deadlock
because no more memory was available.

This fixes the problem by introducing a reservation system which the
multipart transfer uses to ensure it can reserve all the memory for
one chunk before starting.
This commit is contained in:
Nick Craig-Wood
2025-04-22 17:20:03 +01:00
parent 5050f42b8b
commit 117d8d9fdb
4 changed files with 78 additions and 8 deletions

View File

@@ -73,7 +73,7 @@ func UploadMultipart(ctx context.Context, src fs.ObjectInfo, in io.Reader, opt U
for partNum := int64(0); !finished; partNum++ {
// Get a block of memory from the pool and token which limits concurrency.
tokens.Get()
rw := NewRW()
rw := NewRW().Reserve(chunkSize)
if acc != nil {
rw.SetAccounting(acc.AccountRead)
}