Skip to content

Add --checksum param to pipe, put, cp, mirror #5043

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 1 commit into from
Sep 24, 2024
Merged

Add --checksum param to pipe, put, cp, mirror #5043

merged 1 commit into from
Sep 24, 2024

Conversation

klauspost
Copy link
Contributor

@klauspost klauspost commented Sep 24, 2024 •
edited
Loading

Description

Add checksums to pipe, cp, mirror commands.

Add checksum to uploaded object. Values: MD5, CRC32, CRC32C, SHA1 or SHA256. Requires server trailing headers support like AWS, MinIO and possibly others.

--md5 is hidden, but still applied.

Uploads to buckets with object locking will still use MD5.

Bonus: mc stat shows checksum information.

How to test this PR?

Single/multipart upload (mc cp):

λ mc cp 0.txt myminio/testbucket -checksum=SHA256 -q &&mc stat myminio/testbucket/0.txt -q
`e:/gopath/src/github.com/minio/mc/0.txt` -> `myminio/testbucket/0.txt`
Total: 597.67 KiB, Transferred: 597.67 KiB, Speed: 59.64 MiB/s
Name      : 0.txt
Date      : 2024-09-24 16:30:40 CEST
Size      : 598 KiB
ETag      : 5d9003348d6762e91a212b50460b287e
VersionID : c21d90cb-d302-4707-a53d-d3c695bf8d4f
Type      : file
Checksum  : SHA256:X3pWjnxkqONadmZwlOAAF4+jRwLEZnTa/WGKc+s/M4w=
Encryption: SSE-KMS (arn:aws:kms:my-minio-key)
Metadata  :
  Content-Type: text/plain


λ mc cp ab.txt myminio/testbucket -checksum=SHA256 -q &&mc stat myminio/testbucket/ab.txt -q
`e:/gopath/src/github.com/minio/mc/ab.txt` -> `myminio/testbucket/ab.txt`
Total: 381.47 MiB, Transferred: 381.47 MiB, Speed: 239.29 MiB/s
Name      : ab.txt
Date      : 2024-09-24 16:31:10 CEST
Size      : 382 MiB
ETag      : a4831df44d6a59fff21881ccd7d5e86f-24
VersionID : d5cb7670-267a-4e9d-8a8d-a85fd60943c9
Type      : file
Checksum  : SHA256:oKTfpng60iS2rRaWKo8/Ep3wTOHmam8rQZbhzQI1tF0=-24
Encryption: SSE-KMS (arn:aws:kms:my-minio-key)
Metadata  :
  Content-Type: text/plain

(mc put)

λ mc put ab.txt myminio/testbucket -checksum=SHA256 -q &&mc stat myminio/testbucket/ab.txt -q
`e:/gopath/src/github.com/minio/mc/ab.txt` -> `myminio/testbucket/ab.txt`
Total: 381.47 MiB, Transferred: 381.47 MiB, Speed: 758.18 MiB/s
Name      : ab.txt
Date      : 2024-09-24 16:41:42 CEST
Size      : 382 MiB
ETag      : e5c129e03bc360798e6c862fc971f468-24
VersionID : 9ecadd2a-6846-4338-979a-abdbacd3fe95
Type      : file
Checksum  : SHA256:oKTfpng60iS2rRaWKo8/Ep3wTOHmam8rQZbhzQI1tF0=-24
Encryption: SSE-KMS (arn:aws:kms:my-minio-key)
Metadata  :
  Content-Type: text/plain


λ mc put 0.txt myminio/testbucket -checksum=SHA256 -q &&mc stat myminio/testbucket/0.txt -q
`e:/gopath/src/github.com/minio/mc/0.txt` -> `myminio/testbucket/0.txt`
Total: 597.67 KiB, Transferred: 597.67 KiB, Speed: 75.67 MiB/s
Name      : 0.txt
Date      : 2024-09-24 16:42:00 CEST
Size      : 598 KiB
ETag      : b99f4b15c1bd411e130e2b0c98e7df86
VersionID : c8e6b086-d794-4263-8ccd-c52e5cce1749
Type      : file
Checksum  : SHA256:X3pWjnxkqONadmZwlOAAF4+jRwLEZnTa/WGKc+s/M4w=
Encryption: SSE-KMS (arn:aws:kms:my-minio-key)
Metadata  :
  Content-Type: text/plain

(mc pipe):

λ cat 0.txt | mc pipe myminio/testbucket/0.txt -checksum=SHA256&&mc stat myminio/testbucket/0.txt -q
 552.00 KiB / ? [                                                                                                       =                               ] 2.01 MiB/s Name      : 0.txt
Date      : 2024-09-24 16:47:59 CEST
Size      : 598 KiB
ETag      : 062320249d887571850f2daea61b3e1e-1
VersionID : dfb6d042-248b-44fd-8dc7-21273c916666
Type      : file
Checksum  : SHA256:T4uVbgD8TgAJxPDPpgtktYfyVG3AkxP39l/BbWZFFHA=-1
Encryption: SSE-KMS (arn:aws:kms:my-minio-key)
Metadata  :
  Content-Type: text/plain

Types of changes

  • New feature (non-breaking change which adds functionality)

@klauspost
Add --checksum param to pipe, cp, mirror
960bf6c 
Add checksums to pipe, cp, mirror commands.

> Add checksum to uploaded object. Values: MD5, CRC32, CRC32C, SHA1 or SHA256. Requires server trailing headers support like AWS, MinIO and possibly others.

`--md5` is hidden, but still applied.

Uploads to buckets with object locking will still use MD5.

Bonus: `mc stat` shows checksum information.
@klauspost klauspost changed the title Add --checksum param to pipe, cp, mirror Add --checksum param to pipe, put, cp, mirror Sep 24, 2024
@klauspost klauspost mentioned this pull request Sep 24, 2024
Merged
8 tasks
@harshavardhana harshavardhana merged commit be87476 into minio:master Sep 24, 2024
5 checks passed
@klauspost klauspost deleted the add-checksums-to-var branch September 24, 2024 15:52
@djwfyi djwfyi mentioned this pull request Oct 3, 2024
Closed
4 tasks
@Flowm Flowm mentioned this pull request Oct 9, 2024
Closed
klauspost added a commit to klauspost/warp that referenced this pull request Jun 3, 2025
@klauspost
Add optional checksums
7aca373 
Usage is similar to [mc](minio/mc#5043).

> To add a [checksum](https://docs.aws.amazon.com/AmazonS3/latest/userguide/checking-object-integrity.html) to the uploaded objects, use `--checksum` parameter.
The following checksums are supported: `CRC32` (composite), `CRC32-FO` (full object), `CRC32C`, `CRC32-FO`, `CRC32C`, `SHA1`, `SHA256` and `CRC64NVME`.
Adding a checksum will always disable MD5 checksums.
@klauspost klauspost mentioned this pull request Jun 3, 2025
Merged
harshavardhana pushed a commit to minio/warp that referenced this pull request Jun 3, 2025
@klauspost
Add optional checksums (#400)
cee449a 
Usage is similar to [mc](minio/mc#5043).

> To add a [checksum](https://docs.aws.amazon.com/AmazonS3/latest/userguide/checking-object-integrity.html) to the uploaded objects, use `--checksum` parameter.
The following checksums are supported: `CRC32` (composite), `CRC32-FO` (full object), `CRC32C`, `CRC32-FO`, `CRC32C`, `SHA1`, `SHA256` and `CRC64NVME`.
Adding a checksum will always disable MD5 checksums.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@harshavardhana harshavardhana harshavardhana approved these changes

@vadmeste vadmeste Awaiting requested review from vadmeste

@zveinn zveinn Awaiting requested review from zveinn

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@klauspost @harshavardhana