Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Default checksum calculation for httpChecksum to CRC32 #3090

Closed
wants to merge 29 commits into from

Conversation

mullermp
Copy link
Contributor

Defaults CRC32 for httpChecksum implementations and adds CRC64 as a new algorithm.

Copy link
Contributor

@alextwoods alextwoods left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice - looks good overall.

gems/aws-sdk-core/CHANGELOG.md Outdated Show resolved Hide resolved
gems/aws-sdk-s3/lib/aws-sdk-s3/multipart_file_uploader.rb Outdated Show resolved Hide resolved
Copy link

github-actions bot commented Oct 10, 2024

Detected 2 possible performance regressions:

  • aws-sdk-sns.gem_size_kb - z-score regression: 83.0 -> 83.5. Z-score: Infinity
  • aws-sdk-kinesis.gem_size_kb - z-score regression: 73.5 -> 74.0. Z-score: Infinity

Copy link

Detected 4 possible performance regressions:

  • aws-sdk-cloudwatchlogs.gem_size_kb - z-score regression: 112.0 -> 112.5. Z-score: Infinity
  • aws-sdk-s3.get_object_small_allocated_kb - z-score regression: 73.22 -> 75.33. Z-score: 21.94
  • aws-sdk-s3.get_object_large_allocated_kb - z-score regression: 21578.44 -> 21580.56. Z-score: 21.94
  • aws-sdk-s3.put_object_small_allocated_kb - z-score regression: 95.03 -> 97.18. Z-score: 22.04

Copy link

Detected 5 possible performance regressions:

  • aws-sdk-s3.get_object_small_allocated_kb - z-score regression: 59.36 -> 61.18. Z-score: 38.49
  • aws-sdk-s3.get_object_large_allocated_kb - z-score regression: 21564.59 -> 21566.41. Z-score: 39.91
  • aws-sdk-s3.put_object_small_allocated_kb - z-score regression: 78.06 -> 80.26. Z-score: 33.0
  • aws-sdk-s3.put_object_multipart_large_allocated_kb - z-score regression: 32108.0 -> 32112.59. Z-score: 30.49
  • aws-sdk-cloudwatchlogs.gem_size_kb - z-score regression: 112.0 -> 112.5. Z-score: Infinity

Copy link

Detected 5 possible performance regressions:

  • aws-sdk-cloudwatchlogs.gem_size_kb - z-score regression: 112.0 -> 112.5. Z-score: Infinity
  • aws-sdk-s3.get_object_small_allocated_kb - z-score regression: 56.73 -> 58.55. Z-score: 62.82
  • aws-sdk-s3.get_object_large_allocated_kb - z-score regression: 21561.95 -> 21563.78. Z-score: 62.82
  • aws-sdk-s3.put_object_small_allocated_kb - z-score regression: 75.16 -> 77.36. Z-score: 75.64
  • aws-sdk-s3.put_object_multipart_large_allocated_kb - z-score regression: 32096.32 -> 32100.77. Z-score: 41.97

Copy link

Detected 5 possible performance regressions:

  • aws-sdk-cloudwatchlogs.gem_size_kb - z-score regression: 112.0 -> 112.5. Z-score: Infinity
  • aws-sdk-s3.get_object_small_allocated_kb - z-score regression: 63.37 -> 65.42. Z-score: 72.1
  • aws-sdk-s3.get_object_large_allocated_kb - z-score regression: 25664.38 -> 25666.43. Z-score: 72.1
  • aws-sdk-s3.put_object_small_allocated_kb - z-score regression: 80.56 -> 82.92. Z-score: 82.81
  • aws-sdk-s3.put_object_multipart_large_allocated_kb - z-score regression: 32124.11 -> 32129.01. Z-score: 40.28

Copy link

Detected 5 possible performance regressions:

  • aws-sdk-s3.get_object_small_allocated_kb - z-score regression: 70.39 -> 72.61. Z-score: 30.18
  • aws-sdk-s3.get_object_large_allocated_kb - z-score regression: 25671.4 -> 25673.62. Z-score: 30.18
  • aws-sdk-s3.put_object_small_allocated_kb - z-score regression: 90.04 -> 92.41. Z-score: 32.2
  • aws-sdk-s3.put_object_multipart_large_allocated_kb - z-score regression: 21929.59 -> 21934.58. Z-score: 15.81
  • aws-sdk-cloudwatchlogs.gem_size_kb - z-score regression: 112.0 -> 112.5. Z-score: Infinity

Copy link

Detected 5 possible performance regressions:

  • aws-sdk-s3.get_object_small_allocated_kb - z-score regression: 59.65 -> 61.48. Z-score: 67.13
  • aws-sdk-s3.get_object_large_allocated_kb - z-score regression: 21564.88 -> 21566.71. Z-score: 49.55
  • aws-sdk-s3.put_object_small_allocated_kb - z-score regression: 78.75 -> 80.83. Z-score: 34.14
  • aws-sdk-s3.put_object_multipart_large_allocated_kb - z-score regression: 32109.84 -> 32114.53. Z-score: 43.16
  • aws-sdk-cloudwatchlogs.gem_size_kb - z-score regression: 112.0 -> 112.5. Z-score: Infinity

Copy link

Detected 1 possible performance regressions:

  • aws-sdk-cloudwatchlogs.gem_size_kb - z-score regression: 112.0 -> 112.5. Z-score: Infinity

1 similar comment
Copy link

Detected 1 possible performance regressions:

  • aws-sdk-cloudwatchlogs.gem_size_kb - z-score regression: 112.0 -> 112.5. Z-score: Infinity

DOCS

def after_initialize(client)
client.config.request_checksum_calculation =
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I see why we need to do this for the legacy option.... but I am pretty concerned about the customer behavior from this. Doing this effectively means that request_checksum_calculation does not work at all for S3 - that is, its value is ignored no matter where it is set (client, ENV, shared config). I think thats pretty confusing.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, agreed. I think client.config.request_checksum_calculation = 'when_required' unless client.config.compute_checksums satisfies the concern?

User doesn’t set compute checksums:

  • This value defaults to true
  • User can freely set new config to required or optional and it will be honored.

User set compute checksums to false:

  • Setting new config to when_required basically does the same thing
  • Setting to when_optional would maybe be confusing because it would be overwritten, but they have explicitly already chosen a conflicting option of compute checksums false.

I can update the documentation but the "false" case and "when_optional" is probably a less likely use case. However computing checksums in general would make sense to override the when optional anyway.

@mullermp mullermp closed this Nov 19, 2024
@mullermp mullermp deleted the s3-crc32 branch November 19, 2024 18:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants