Skip to content

Incorrect multipart threshold #1994

@jasoncwik

Description

@jasoncwik

Currently, if PutObject is executed with exactly partSize / minPartSize, it will execute a multipart upload with exactly one part. This is wasteful and causes performance issues, especially over the WAN, due to multiple round-trip operations.

Example:

  1. Create object exactly 16MiB
/tmp/minio-examples/examples/s3 $ dd if=/dev/random of=my-testfile bs=1M count=16
16+0 records in
16+0 records out
16777216 bytes (17 MB, 16 MiB) copied, 0.0712859 s, 235 MB/s
  1. Upload with sample PutObject call
/tmp/minio-examples/examples/s3 $ ls -l my-testfile 
-rw-rw-r-- 1 jcwik jcwik 16777216 Aug 28 09:27 my-testfile

/tmp/minio-examples/examples/s3 $ ./putobject | tee put1.txt
<snip>
2024/08/28 10:29:42 Uploaded my-objectname  of size:  16777216 Successfully in  421.786748ms

You can see there are 3 calls made for the upload, creating a MPU with a single part.

/tmp/minio-examples/examples/s3 $ grep -A 1 START-HT put1.txt
---------START-HTTP---------
GET /testbucket/?location= HTTP/1.1
--
---------START-HTTP---------
POST /testbucket/my-objectname?uploads= HTTP/1.1
--
---------START-HTTP---------
PUT /testbucket/my-objectname?partNumber=1&uploadId=ZGE0YzdiNmQtNzNhYS00MzlhLTgyMzEtZThhMjA2MGM3ZThkLmRjNGUwODUzLTcxY2YtNDhlYy1hZTJkLTAxZTYyNDcyNjYyNngxNzI0ODU5MDE5MjcwOTE2NzI3 HTTP/1.1
--
---------START-HTTP---------
POST /testbucket/my-objectname?uploadId=ZGE0YzdiNmQtNzNhYS00MzlhLTgyMzEtZThhMjA2MGM3ZThkLmRjNGUwODUzLTcxY2YtNDhlYy1hZTJkLTAxZTYyNDcyNjYyNngxNzI0ODU5MDE5MjcwOTE2NzI3 HTTP/1.1

I have a PR, will post in a few minutes.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions