Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Multi-part s3 uploads fail when using checksum #6793

Closed
avantgardnerio opened this issue Nov 25, 2024 · 1 comment · Fixed by #6794
Closed

Multi-part s3 uploads fail when using checksum #6793

avantgardnerio opened this issue Nov 25, 2024 · 1 comment · Fixed by #6794
Labels
bug object-store Object Store Interface

Comments

@avantgardnerio
Copy link
Contributor

avantgardnerio commented Nov 25, 2024

Describe the bug

with_checksum_algorithm() fails when used in with put_multipart_opts() on large files

To Reproduce

  1. Set standard AWS env vars
  2. Run test
  3. observe:
message: "Checksum Type mismatch occurred, expected checksum Type: null, actual checksum Type: sha256

Expected behavior

It should write the file

Additional context

See PR for failing test

@thinkharderdev
Copy link
Contributor

thinkharderdev commented Nov 25, 2024

So there are a couple of issues with how multi-part upload is implemented:

  1. The initial PUT request to CreateMultipartUpload must specify the checksum algo or else calling PutPart with the checksum will fail (eg set x-amz-checksum-algorithm: SHA256 on the request)
  2. When calling CompleteMultipartUpload we need to send all the part checksums in the request (see `https://docs.aws.amazon.com/AmazonS3/latest/API/API_CompleteMultipartUpload.html)

So this means that we need to keep the checksums in the PartId for when we complete the upload

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug object-store Object Store Interface
Projects
None yet
3 participants