High memory overhead when patching aws-sdk

[barrucadu]

Hello,

We recently added aws-xray-sdk to a bunch of applications, and those where we patch aws_sdk started using significantly more memory than usual. Here is how it's configured:

# if aws-sdk is loaded, we want to instrument that too
patch = Gem.loaded_specs.has_key?('aws-sdk-core') ?
          %I[aws_sdk net_http] : %I[net_http]

# if there isn't a name set, attempting to record a segment will
# throw an error
govuk_app_name = ENV['GOVUK_APP_NAME']
name = govuk_app_name.blank? ? 'xray' : govuk_app_name

XRay.recorder.configure(
  name: name,
  patch: patch,
  context_missing: 'LOG_ERROR',
  sampling_rules: {
    version: 1,
    default: {
      'fixed_target': ENV.fetch('XRAY_SAMPLE_TARGET', 0).to_i,
      'rate': ENV.fetch('XRAY_SAMPLE_RATE', 0.01).to_f,
    },
    rules: [],
  },
)

I don't think this is a memory leak, but an overhead incurred when the patching is performed. Here is the memory usage of our publishing-api application, which shot up dramatically when the X-Ray configuration was deployed:

The application doesn't actually use any AWS stuff, but it depends on some AWS gems because there are a couple of rake tasks to import and export data to S3. So despite the application itself not doing any AWS stuff, the memory usage is still dramatically increased.

The reason I think this is a problem with the aws_sdk patching is because the same problem does not occur when only net_http is patched. The same application, with only net_http patched, looks like this:

When the change was deployed, the memory usage plummeted. Towards the end of the graph, a branch which enabled patching of aws-sdk was deployed, and the memory usage goes up. This seems like too much of a coincidence for it not to be the aws-sdk patching at fault.

There's some more details, including graphs for a second application, here: alphagov/govuk_app_config#61

[haotianw465]

Thank you for reporting the issue and all the deep dive. We will profile the aws_sdk patching as soon as possible. It might be related to the default patching behavior where it adds X-Ray plugin to all loaded AWS Clients https://github.com/aws/aws-xray-sdk-ruby/blob/master/lib/aws-xray-sdk/facets/aws_sdk.rb#L130. Could you point me to your dependency config on AWS SDK related gems so the profiling at our end is closer to your use case?

[barrucadu]

We're using aws-sdk ~> 3. Here's the Gemfile and Gemfile.lock.

Here's the configuration, in config/initializers/aws.rb:

require 'aws-sdk'

Aws.config.update(
  logger: ::Rails.logger,
  region: ENV["S3_EXPORT_REGION"] || "eu-west-1",
  credentials: Aws::Credentials.new(ENV["EVENT_LOG_AWS_ACCESS_ID"] || 'id', ENV["EVENT_LOG_AWS_SECRET_KEY"] || 'key')
)

[haotianw465]

AWS Ruby SDK v3 is fully modularized and it allows you to require the gem you needed (AWS services you are using). This reduces your bundle size and client loaded in memory. It also gives you flexibility of update individual gems. https://aws.amazon.com/sdk-for-ruby/

As you can see in the patcher link, it tries to require all the possible gems and patch those clients if no explicit gems are specified. If you only require aws-sdk-s3, then only the s3 client is loaded and patched, which should reduce the memory usage quite a bit.

Do you mind changing aws-sdk to aws-sdk-s3 in your Gemfile and see if there is an improvement on memory usage? Or is there any specific reason that you need to use aws-sdk and requiring service gems individually doesn't fit your use case?

[barrucadu]

I've changed it to only require aws-sdk-s3 and once again patch aws_sdk (https://github.com/alphagov/publishing-api/compare/msw/xrayperiments). Memory usage still jumps up by quite a bit on deploy:

It's hard to tell if it's better or not, it looks like a similar amount as the jump towards the end of my second graph.