Aws::S3::Object::Collection#batch_delete! raises Aws::S3::Errors::MissingContentMD5 error on minio after updating aws-sdk-s3 to 1.178.0

[tatsuyafw]

Describe the bug

Hi,

I'm using minio and the aws-sdk-ruby gem.
After updating the aws-sdk-ruby gem from 1.177.0 to 1.178.0, Aws::S3::Object::Collection#batch_delete! raises an Aws::S3::Errors::MissingContentMD5 error as below:

aws-sdk-s3 1.178.0

s3_resource = Aws::S3::Resource.new(
  access_key_id: '<secret>',
  secret_access_key: '<secret>',
  endpoint: 'http://localhost:9000', # Minio
  region: 'ap-northeast-1',
  force_path_style: true
)

s3 = s3_resource.bucket('sandbox')

s3.objects.batch_delete!
#=> Missing required header for this request: Content-Md5. (Aws::S3::Errors::MissingContentMD5)

puts Gem.loaded_specs["aws-sdk-s3"].version.to_s
#=> 1.178.0

aws-sdk-s3 1.177.0

s3.objects.batch_delete!
#=> works fine

puts Gem.loaded_specs["aws-sdk-s3"].version.to_s 
#=> 1.177.0

Regression Issue

• Select this option if this issue appears to be a regression.

Expected Behavior

The expected behavior seems that the Aws::S3::Errors::MissingContentMD5 error is not be raised.

Current Behavior

Aws::S3::Object::Collection#batch_delete! raises an Aws::S3::Errors::MissingContentMD5 error.

Reproduction Steps

s3_resource = Aws::S3::Resource.new(
  access_key_id: '<secret>',
  secret_access_key: '<secret>',
  endpoint: 'http://localhost:9000', # Minio
  region: 'ap-northeast-1',
  force_path_style: true
)

s3 = s3_resource.bucket('sandbox')

s3.objects.batch_delete!
#=> Missing required header for this request: Content-Md5. (Aws::S3::Errors::MissingContentMD5)

Possible Solution

No response

Additional Information/Context

Upon examining the minio server logs, I discovered that the Content-Md5 HTTP header, previously sent by clients in version 1.177.0, was absent in version 1.178.0.

minio server logs

# aws-sdk-s3 1.177.0
localhost:9000 [REQUEST s3.DeleteMultipleObjects] [2025-01-16T20:02:25.717] [Client IP: 172.18.0.1]
localhost:9000 POST /sample?delete
localhost:9000 Proto: HTTP/1.1
localhost:9000 Host: localhost:9000
localhost:9000 User-Agent: aws-sdk-ruby3/3.214.1 ua/2.1 api/s3#1.177.0 os/macos#24 md/arm64 lang/ruby#3.3.0 md/3.3.0 m/A,N,D
// snip
localhost:9000 Content-Md5: GSHZcUfDwFe/xxbd148DWA==
// snip

# aws-sdk-s3 1.178.0
localhost:9000 [REQUEST s3.DeleteMultipleObjects] [2025-01-16T19:58:11.700] [Client IP: 172.18.0.1]
localhost:9000 POST /sample?delete
localhost:9000 Proto: HTTP/1.1
localhost:9000 Host: localhost:9000
localhost:9000 User-Agent: aws-sdk-ruby3/3.216.0 ua/2.1 api/s3#1.178.0 os/macos#24 md/arm64 lang/ruby#3.3.0 md/3.3.0 m/A,N,Z,b,U,D
localhost:9000 X-Amz-Checksum-Crc32: gcia8A==
localhost:9000 X-Amz-Sdk-Checksum-Algorithm: CRC32
localhost:9000 Accept: */*
localhost:9000 Accept-Encoding:
localhost:9000 Content-Length: 103
localhost:9000 Content-Type: application/xml
// snip

In commit df86be6, specifically the changes made to gems/aws-sdk-core/lib/aws-sdk-core/plugins/http_checksum.rb, it appears that the Content-Md5 HTTP header is no longer being added in certain conditions. Is this an intentional change?

df86be6#diff-2b60978428d604b66d702d6b0dd0580485061cac7334f236979177c4fe7659c4R14

Gem name ('aws-sdk', 'aws-sdk-resources' or service gems like 'aws-sdk-s3') and its version

aws-sdk-s3 1.178.0

Environment details (Version of Ruby, OS environment)

ruby 3.3.0 (2023-12-25 revision 5124f9ac75) [arm64-darwin24]

[mullermp]

Unfortunately this is expected behavior with a recent change by S3 and 3rd party products such as min.io #3166

[mullermp]

You have a few options -

option 1 - do not update s3 until min.io supports other checksum types that S3 supports such as crc32, sha256, etc.

option 2 - If you want to calculate MD5 for some integrity checks, you may have to do it yourself, and pass it to APIs, or if there is no such content_md5 member, you can use a custom plugin. Here is how you can do that:

class ContentMD5 < Seahorse::Client::Plugin
  class Handler < Seahorse::Client::Handler
    def call(context)
      body = context.http_request.body
      context.http_request.headers['Content-Md5'] = calculate_md5(body)
      body.rewind
      @handler.call(context)
    end

    def calculate_md5(body)
      # ..
    end
  end

  handler(Handler, step: :sign)
end

Aws::S3::Client.add_plugin(ContentMD5)

You can probably pre-compute md5 and pass it through context or probably insert this calculation somewhere else in the handler stack.

[tatsuyafw]

@mullermp Thank you very much for your prompt response! Your comments have been very helpful.

I am currently using MinIO only in the development and test environments of my Ruby on Rails project, and I am using AWS S3 in the production environment.

Therefore, I plan to use a plugin that adds the Content-MD5 header only in the development and test environments, and remove the plugin when the MinIO server supports other checksums.

I've created a plugin as described below and confirmed that the DeleteObjects operation works fine.

class AwsS3DeleteObjectsPlugin < Seahorse::Client::Plugin
  class Handler < Seahorse::Client::Handler
    TARGET_OPERATION_NAME = 'DeleteObjects'.freeze
    HTTP_HEADER_NAME = 'Content-Md5'.freeze

    def call(context)
      if context.operation.name == TARGET_OPERATION_NAME
        body = context.http_request.body
        context.http_request.headers[HTTP_HEADER_NAME] = calculate_md5(body)
      end
      @handler.call(context)
    end

    private

    def calculate_md5(body)
      Base64.encode64(OpenSSL::Digest::MD5.digest(body.string))
    end
  end

  handler(Handler, step: :sign)
end

if Rails.env.development? || Rails.env.test?
  Aws::S3::Client.add_plugin(AwsS3DeleteObjectsPlugin)
end

[github-actions]

This issue is now closed. Comments on closed issues are hard for our team to see.
If you need more assistance, please open a new issue that references this one.

[mullermp]

Yes. One improvement is that you can pass operations to handler() instead of checking yourself.

handler(Handler, step: :sign, operations: [:delete_objects])

[tatsuyafw]

@mullermp Thank you for your advice!