Skip to content

Large multipart uploads with aws cli to GCS fail #330

@dantman

Description

@dantman

I was using a docker image (mysql-backup) that uploads to a "S3" backend using the aws cli. Trying to use it with s3proxy connected to GCP as the upload destination fails with the following error.

An error occurred (BadDigest) when calling the CompleteMultipartUpload operation

More details from the log (includes the shell commands executed and full cli response)

+ AWS_ENDPOINT_OPT='--endpoint-url http://cloud-backups-s3proxy.default.svc.cluster.local'
+ aws --endpoint-url http://cloud-backups-s3proxy.default.svc.cluster.local s3 cp /tmp/backups/db_backup_2020-08-24T20:46:05Z.tgz s3://cloud-backups/db/db_backup_2020-08-24T20:46:05Z.tgz
Complupload failed: tmp/backups/db_backup_2020-08-24T20:46:05Z.tgz to s3://cloud-backups/db/db_backup_2020-08-24T20:46:05Z.tgz An error occurred (BadDigest) when calling the CompleteMultipartUpload operation (reached max retries: 4): Bad Request

I assume a similar error could be triggered by configuring s3 to use GCP, trying to use the aws s3 cli to to and upload, and using the multipart_threshold to force the cli to do multipart uploads for smaller files.

I also presume GCP and Amazon have different interpretations of how hash digests of multipart uploads should work.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions