Skip to content

Commit

Permalink
Move create defaults to transfer manager
Browse files Browse the repository at this point in the history
  • Loading branch information
mullermp committed Oct 14, 2024
1 parent 2b20f4c commit 2a94537
Show file tree
Hide file tree
Showing 8 changed files with 100 additions and 117 deletions.
4 changes: 2 additions & 2 deletions gems/aws-sdk-core/CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -1,9 +1,9 @@
Unreleased Changes
------------------

* Feature - Always calculate request checksums for operations that support or require it. Supported config options are `WHEN_SUPPORTED` and `WHEN_REQUIRED`. The default value is `WHEN_SUPPORTED`. This option is configured in code with `:request_checksum_calculation`, in the shared config file as `request_checksum_calculation`, and in the ENV as `ENV['AWS_REQUEST_CHECKSUM_CALCULATION']`.
* Feature - Always calculate request checksums for operations that support or require it. Supported config options are `when_supported` and `when_required`. The default value is `when_supported`. This option is configured in code with `:request_checksum_calculation`, in the shared config file as `request_checksum_calculation`, and in the ENV as `ENV['AWS_REQUEST_CHECKSUM_CALCULATION']`.

* Feature - Always validate response checksums for operations that support or require it. Supported config options are `WHEN_SUPPORTED` and `WHEN_REQUIRED`. The default value is `WHEN_SUPPORTED`. This option is configured in code with `:response_checksum_validation`, in the shared config file as `response_checksum_validation`, and in the ENV as `ENV['AWS_response_checksum_validation']`.
* Feature - Always validate response checksums for operations that support or require it. Supported config options are `when_supported` and `when_required`. The default value is `when_supported`. This option is configured in code with `:response_checksum_validation`, in the shared config file as `response_checksum_validation`, and in the ENV as `ENV['AWS_response_checksum_validation']`.

3.209.1 (2024-09-25)
------------------
Expand Down
50 changes: 25 additions & 25 deletions gems/aws-sdk-core/lib/aws-sdk-core/plugins/checksum_algorithm.rb
Original file line number Diff line number Diff line change
Expand Up @@ -38,16 +38,16 @@ class ChecksumAlgorithm < Seahorse::Client::Plugin
DEFAULT_CHECKSUM = 'CRC32'

option(:request_checksum_calculation,
doc_default: 'WHEN_SUPPORTED',
doc_default: 'when_supported',
doc_type: 'String',
docstring: <<~DOCS) do |cfg|
Determines when a checksum will be calculated for request payloads. Values are:
* `WHEN_SUPPORTED` - (default) When set, a checksum will be
* `when_supported` - (default) When set, a checksum will be
calculated for all request payloads of operations modeled with the
`httpChecksum` trait where `requestChecksumRequired` is `true` and/or a
`requestAlgorithmMember` is modeled.
* `WHEN_REQUIRED` - When set, a checksum will only be calculated for
* `when_required` - When set, a checksum will only be calculated for
request payloads of operations modeled with the `httpChecksum` trait where
`requestChecksumRequired` is `true` or where a requestAlgorithmMember
is modeled and supplied.
Expand All @@ -56,16 +56,16 @@ class ChecksumAlgorithm < Seahorse::Client::Plugin
end

option(:response_checksum_validation,
doc_default: 'WHEN_SUPPORTED',
doc_default: 'when_supported',
doc_type: 'String',
docstring: <<~DOCS) do |cfg|
Determines when checksum validation will be performed on response payloads. Values are:
* `WHEN_SUPPORTED` - (default) When set, checksum validation is performed on all
* `when_supported` - (default) When set, checksum validation is performed on all
response payloads of operations modeled with the `httpChecksum` trait where
`responseAlgorithms` is modeled, except when no modeled checksum algorithms
are supported.
* `WHEN_REQUIRED` - When set, checksum validation is not performed on
* `when_required` - When set, checksum validation is not performed on
response payloads of operations unless the checksum algorithm is supported and
the `requestValidationModeMember` member is set to `ENABLED`.
DOCS
Expand Down Expand Up @@ -102,27 +102,27 @@ def trailer_length(algorithm, location_name)
def resolve_request_checksum_calculation(cfg)
mode = ENV['AWS_REQUEST_CHECKSUM_CALCULATION'] ||
Aws.shared_config.request_checksum_calculation(profile: cfg.profile) ||
'WHEN_SUPPORTED'
mode = mode.upcase
unless %w[WHEN_SUPPORTED WHEN_REQUIRED].include?(mode)
'when_supported'
mode = mode.downcase
unless %w[when_supported when_required].include?(mode)
raise ArgumentError,
'expected :request_checksum_calculation or' \
" ENV['AWS_REQUEST_CHECKSUM_CALCULATION'] to be " \
'`WHEN_SUPPORTED` or `WHEN_REQUIRED`.'
'`when_supported` or `when_required`.'
end
mode
end

def resolve_response_checksum_validation(cfg)
mode = ENV['AWS_response_checksum_validation'] ||
Aws.shared_config.response_checksum_validation(profile: cfg.profile) ||
'WHEN_SUPPORTED'
mode = mode.upcase
unless %w[WHEN_SUPPORTED WHEN_REQUIRED].include?(mode)
'when_supported'
mode = mode.downcase
unless %w[when_supported when_required].include?(mode)
raise ArgumentError,
'expected :response_checksum_validation or' \
" ENV['AWS_response_checksum_validation'] to be " \
'`WHEN_SUPPORTED` or `WHEN_REQUIRED`.'
'`when_supported` or `when_required`.'
end
mode
end
Expand Down Expand Up @@ -160,7 +160,7 @@ def call(context)
context[:http_checksum] ||= {}

# Set validation mode to enabled when supported.
if context.config.response_checksum_validation == 'WHEN_SUPPORTED'
if context.config.response_checksum_validation == 'when_supported'
enable_request_validation_mode(context)
end

Expand Down Expand Up @@ -221,19 +221,19 @@ def with_metrics(config, algorithm, &block)

def add_request_config_metric(config, metrics)
case config.request_checksum_calculation
when 'WHEN_SUPPORTED'
metrics << 'FLEXIBLE_CHECKSUMS_REQ_WHEN_SUPPORTED'
when 'WHEN_REQUIRED'
metrics << 'FLEXIBLE_CHECKSUMS_REQ_WHEN_REQUIRED'
when 'when_supported'
metrics << 'FLEXIBLE_CHECKSUMS_REQ_when_supported'
when 'when_required'
metrics << 'FLEXIBLE_CHECKSUMS_REQ_when_required'
end
end

def add_response_config_metric(config, metrics)
case config.response_checksum_validation
when 'WHEN_SUPPORTED'
metrics << 'FLEXIBLE_CHECKSUMS_RES_WHEN_SUPPORTED'
when 'WHEN_REQUIRED'
metrics << 'FLEXIBLE_CHECKSUMS_RES_WHEN_REQUIRED'
when 'when_supported'
metrics << 'FLEXIBLE_CHECKSUMS_RES_when_supported'
when 'when_required'
metrics << 'FLEXIBLE_CHECKSUMS_RES_when_required'
end
end

Expand Down Expand Up @@ -275,12 +275,12 @@ def operation_response_algorithms(context)
def checksum_required?(context)
(http_checksum = context.operation.http_checksum) &&
(checksum_required = http_checksum['requestChecksumRequired']) &&
(checksum_required && context.config.request_checksum_calculation == 'WHEN_REQUIRED')
(checksum_required && context.config.request_checksum_calculation == 'when_required')
end

def checksum_optional?(context)
context.operation.http_checksum &&
context.config.request_checksum_calculation == 'WHEN_SUPPORTED'
context.config.request_checksum_calculation == 'when_supported'
end

def checksum_provided_as_header?(headers)
Expand Down
50 changes: 25 additions & 25 deletions gems/aws-sdk-core/spec/aws/plugins/checksum_algorithm_spec.rb
Original file line number Diff line number Diff line change
Expand Up @@ -101,58 +101,58 @@ module Plugins
describe 'request_checksum_calculation' do
it 'is configured to always compute by default' do
expect(client.config.request_checksum_calculation)
.to eq('WHEN_SUPPORTED')
.to eq('when_supported')
end

it 'can be configured using shared config' do
allow_any_instance_of(Aws::SharedConfig)
.to receive(:request_checksum_calculation)
.and_return('WHEN_REQUIRED')
.and_return('when_required')
expect(client.config.request_checksum_calculation)
.to eq('WHEN_REQUIRED')
.to eq('when_required')
end

it 'can be configured using ENV with precedence over shared config' do
allow_any_instance_of(Aws::SharedConfig)
.to receive(:request_checksum_calculation)
.and_return('WHEN_SUPPORTED')
ENV['AWS_REQUEST_CHECKSUM_CALCULATION'] = 'WHEN_REQUIRED'
.and_return('when_supported')
ENV['AWS_REQUEST_CHECKSUM_CALCULATION'] = 'when_required'
expect(client.config.request_checksum_calculation)
.to eq('WHEN_REQUIRED')
.to eq('when_required')
end

it 'raises when request_checksum_calculation is not valid' do
ENV['AWS_REQUEST_CHECKSUM_CALCULATION'] = 'peccy'
expect { client }.to raise_error(ArgumentError, /WHEN_SUPPORTED/)
expect { client }.to raise_error(ArgumentError, /when_supported/)
end
end

describe 'response_checksum_validation' do
it 'is configured to always verify by default' do
expect(client.config.response_checksum_validation)
.to eq('WHEN_SUPPORTED')
.to eq('when_supported')
end

it 'can be configured using shared config' do
allow_any_instance_of(Aws::SharedConfig)
.to receive(:response_checksum_validation)
.and_return('WHEN_REQUIRED')
.and_return('when_required')
expect(client.config.response_checksum_validation)
.to eq('WHEN_REQUIRED')
.to eq('when_required')
end

it 'can be configured using ENV with precedence over shared config' do
allow_any_instance_of(Aws::SharedConfig)
.to receive(:response_checksum_validation)
.and_return('WHEN_SUPPORTED')
ENV['AWS_response_checksum_validation'] = 'WHEN_REQUIRED'
.and_return('when_supported')
ENV['AWS_response_checksum_validation'] = 'when_required'
expect(client.config.response_checksum_validation)
.to eq('WHEN_REQUIRED')
.to eq('when_required')
end

it 'raises when response_checksum_validation is not valid' do
ENV['AWS_response_checksum_validation'] = 'peccy'
expect { client }.to raise_error(ArgumentError, /WHEN_SUPPORTED/)
expect { client }.to raise_error(ArgumentError, /when_supported/)
end
end

Expand Down Expand Up @@ -311,16 +311,16 @@ module Plugins
context 'when checksums are not required' do
let(:request_checksum_required) { false }

it 'WHEN_SUPPORTED; no algorithm; includes a checksum' do
it 'when_supported; no algorithm; includes a checksum' do
resp = client.http_checksum_operation(checksum_algorithm: 'CRC32')
expect(resp.context.http_request.headers['x-amz-checksum-crc32'])
.to eq('AAAAAA==')
end

it 'WHEN_REQUIRED; no algorithm; does not include a checksum' do
it 'when_required; no algorithm; does not include a checksum' do
client = checksum_client.new(
stub_responses: true,
request_checksum_calculation: 'WHEN_REQUIRED'
request_checksum_calculation: 'when_required'
)
resp = client.http_checksum_operation
expect(resp.context.http_request.headers['x-amz-checksum-crc32'])
Expand All @@ -331,16 +331,16 @@ module Plugins
context 'when checksums are required' do
let(:request_checksum_required) { true }

it 'WHEN_SUPPORTED; no algorithm; includes a checksum' do
it 'when_supported; no algorithm; includes a checksum' do
resp = client.http_checksum_operation
expect(resp.context.http_request.headers['x-amz-checksum-crc32'])
.to eq('AAAAAA==')
end

it 'WHEN_REQUIRED; no algorithm; includes a checksum' do
it 'when_required; no algorithm; includes a checksum' do
client = checksum_client.new(
stub_responses: true,
request_checksum_calculation: 'WHEN_REQUIRED'
request_checksum_calculation: 'when_required'
)
resp = client.http_checksum_operation
expect(resp.context.http_request.headers['x-amz-checksum-crc32'])
Expand All @@ -362,28 +362,28 @@ def stub_client(client)
)
end

it 'WHEN_SUPPORTED; not ENABLED; validates the checksum' do
it 'when_supported; not ENABLED; validates the checksum' do
stub_client(client)
resp = client.http_checksum_operation
expect(resp.context[:http_checksum][:validated]).to eq('CRC32')
# This needs to be set by the plugin in this case
expect(resp.context.params[:validation_mode]).to eq('ENABLED')
end

it 'WHEN_REQUIRED; not ENABLED; does not validate the checksum' do
it 'when_required; not ENABLED; does not validate the checksum' do
client = checksum_client.new(
stub_responses: true,
response_checksum_validation: 'WHEN_REQUIRED'
response_checksum_validation: 'when_required'
)
stub_client(client)
resp = client.http_checksum_operation
expect(resp.context[:http_checksum][:validated]).to be_nil
end

it 'WHEN_REQUIRED; ENABLED; validates the checksum' do
it 'when_required; ENABLED; validates the checksum' do
client = checksum_client.new(
stub_responses: true,
response_checksum_validation: 'WHEN_REQUIRED'
response_checksum_validation: 'when_required'
)
stub_client(client)
resp = client.http_checksum_operation(validation_mode: 'ENABLED')
Expand Down
35 changes: 7 additions & 28 deletions gems/aws-sdk-s3/lib/aws-sdk-s3/client.rb
Original file line number Diff line number Diff line change
Expand Up @@ -40,7 +40,6 @@
require 'aws-sdk-s3/plugins/arn.rb'
require 'aws-sdk-s3/plugins/bucket_dns.rb'
require 'aws-sdk-s3/plugins/bucket_name_restrictions.rb'
require 'aws-sdk-s3/plugins/checksum_algorithm.rb'
require 'aws-sdk-s3/plugins/dualstack.rb'
require 'aws-sdk-s3/plugins/expect_100_continue.rb'
require 'aws-sdk-s3/plugins/express_session_auth.rb'
Expand All @@ -53,6 +52,7 @@
require 'aws-sdk-s3/plugins/s3_host_id.rb'
require 'aws-sdk-s3/plugins/s3_signer.rb'
require 'aws-sdk-s3/plugins/sse_cpk.rb'
require 'aws-sdk-s3/plugins/skip_whole_multipart_get_checksums.rb'
require 'aws-sdk-s3/plugins/streaming_retry.rb'
require 'aws-sdk-s3/plugins/url_encoded_keys.rb'
require 'aws-sdk-core/plugins/event_stream_configuration.rb'
Expand Down Expand Up @@ -111,7 +111,6 @@ class Client < Seahorse::Client::Base
add_plugin(Aws::S3::Plugins::ARN)
add_plugin(Aws::S3::Plugins::BucketDns)
add_plugin(Aws::S3::Plugins::BucketNameRestrictions)
add_plugin(Aws::S3::Plugins::ChecksumAlgorithm)
add_plugin(Aws::S3::Plugins::Dualstack)
add_plugin(Aws::S3::Plugins::Expect100Continue)
add_plugin(Aws::S3::Plugins::ExpressSessionAuth)
Expand All @@ -124,6 +123,7 @@ class Client < Seahorse::Client::Base
add_plugin(Aws::S3::Plugins::S3HostId)
add_plugin(Aws::S3::Plugins::S3Signer)
add_plugin(Aws::S3::Plugins::SseCpk)
add_plugin(Aws::S3::Plugins::SkipWholeMultipartGetChecksums)
add_plugin(Aws::S3::Plugins::StreamingRetry)
add_plugin(Aws::S3::Plugins::UrlEncodedKeys)
add_plugin(Aws::Plugins::EventStreamConfiguration)
Expand Down Expand Up @@ -240,9 +240,11 @@ class Client < Seahorse::Client::Base
# will use the Client Side Monitoring Agent Publisher.
#
# @option options [Boolean] :compute_checksums (true)
# This option is deprecated. Please use `:request_checksum_calculation`
# instead. When `true`, `request_checksum_calculation` is set to `WHEN_SUPPORTED`,
# and if `false` it is set to `WHEN_REQUIRED`.
# When `true` a MD5 checksum will be computed and sent in the Content Md5
# header for :put_object and :upload_part. When `false`, MD5 checksums
# will not be computed for these operations. Checksums are still computed
# for operations requiring them. Checksum errors returned by Amazon S3 are
# automatically retried up to `:retry_limit` times.
#
# @option options [Boolean] :convert_params (true)
# When `true`, an attempt is made to coerce request parameters into
Expand Down Expand Up @@ -338,18 +340,6 @@ class Client < Seahorse::Client::Base
# Used when loading credentials from the shared credentials file
# at HOME/.aws/credentials. When not specified, 'default' is used.
#
# @option options [String] :request_checksum_calculation ("WHEN_SUPPORTED")
# Determines when a checksum will be calculated for request payloads. Values are:
#
# * `WHEN_SUPPORTED` - (default) When set, a checksum will be
# calculated for all request payloads of operations modeled with the
# `httpChecksum` trait where `requestChecksumRequired` is `true` and/or a
# `requestAlgorithmMember` is modeled.
# * `WHEN_REQUIRED` - When set, a checksum will only be calculated for
# request payloads of operations modeled with the `httpChecksum` trait where
# `requestChecksumRequired` is `true` or where a requestAlgorithmMember
# is modeled and supplied.
#
# @option options [Integer] :request_min_compression_size_bytes (10240)
# The minimum size in bytes that triggers compression for request
# bodies. The value must be non-negative integer value between 0
Expand All @@ -360,17 +350,6 @@ class Client < Seahorse::Client::Base
# where server-side-encryption is used with customer-provided keys.
# This should only be disabled for local testing.
#
# @option options [String] :response_checksum_validation ("WHEN_SUPPORTED")
# Determines when checksum validation will be performed on response payloads. Values are:
#
# * `WHEN_SUPPORTED` - (default) When set, checksum validation is performed on all
# response payloads of operations modeled with the `httpChecksum` trait where
# `responseAlgorithms` is modeled, except when no modeled checksum algorithms
# are supported.
# * `WHEN_REQUIRED` - When set, checksum validation is not performed on
# response payloads of operations unless the checksum algorithm is supported and
# the `requestValidationModeMember` member is set to `ENABLED`.
#
# @option options [Proc] :retry_backoff
# A proc or lambda used for backoff. Defaults to 2**retries * retry_base_delay.
# This option is only used in the `legacy` retry mode.
Expand Down
2 changes: 1 addition & 1 deletion gems/aws-sdk-s3/lib/aws-sdk-s3/file_downloader.rb
Original file line number Diff line number Diff line change
Expand Up @@ -28,7 +28,7 @@ def download(destination, options = {})
@chunk_size = options[:chunk_size]
@params = {
bucket: options[:bucket],
key: options[:key],
key: options[:key]
}
@params[:version_id] = options[:version_id] if options[:version_id]
@on_checksum_validated = options[:on_checksum_validated]
Expand Down
Loading

0 comments on commit 2a94537

Please sign in to comment.