aws_s3_bucket_object eventual consistency error after object creation

[jpgu-epam]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Terraform Version

0.12.20

Aws Provider Version

2.54

Affected Resource(s)

• aws_s3_bucket_object

Terraform Configuration Files

resource "aws_s3_bucket_object" "foo-archive" {
  key                    = "foo.sh"
  bucket                 = aws_s3_bucket.foo_bucket.bucket
  source                 = foo.path
  server_side_encryption = "aws:kms"
}

Expected Behavior

Resource creation is successful.

Actual Behavior

aws_s3_bucket_object.foo: Failed to get object tags (bucket: some-bucket, key: /foo): NoSuchKey: The specified key does not

As is generally the case with eventual consistency issues, this only happens sometimes.

Steps to Reproduce

Just define a standard S3 bucket object like so and run apply:

resource "aws_s3_bucket_object" "foo" {
bucket = "some-bucket"
key = "/foo"
source = "${path.module}/foo"
}

Eventually, you get:

Error: error listing tags for S3 Bucket (foo-test-1f0mzi-foo-cache) Object (/foo): NoSuchKey: The specified key does not exist.
	status code: 404

References

• aws_s3_bucket_object eventual consistency error after object creation #5841

[borfig]

It looks like the usual "eventual consistency" in AWS API.
For example, in my CloudTrail log, I see that the between the bucket creation and probing its tagging there were 0.014 seconds... a retry should be added
(I have seen similar AWS API behaviour in other services as well)

[rjmcnamara]

Is there a workaround for this? I'm seeing the issue with v3.14.0

In my case, I'm creating 4 folder objects and fairly often 1 or 2 of them will fail with "NoSuchKey: The specified key does not exist". The ones that do succeed report "Creation complete after 0s"

I have not tried experimenting with max_retries yet, but given that the default is 25, I have my doubts that increasing that would help.

[novekm]

Are there any updates for this? I am running into the same issue with the latest version of the AWS provider, and using the newer (recommended) aws_s3_object resource. @justinretzolk do you have any ideas what may be causing this? Considering just using local-exec provisioner within a null_resource in the meantime, but would prefer to avoid that since I need to reference the object elsewhere in the project.

[novekm]

For anyone else still having this issue, this is the temporary workaround I had to go with:

# Temporary workaround until this GitHub issue on aws_s3_object is resolved: https://github.com/hashicorp/terraform-provider-aws/issues/12652
resource "null_resource" "put_s3_object" {
  # Will only trigger this resource to re-run if changes are made to the Dockerfile
  triggers = {
    src_hash = data.archive_file.streamlit_assets.output_sha
  }

  # Put .zip file for Streamlit App Assets in S3 Bucket
  provisioner "local-exec" {
    command = "aws s3 cp ${var.app_name}-assets.zip s3://${aws_s3_bucket.streamlit_s3_bucket.id}/${var.app_name}-assets.zip"

  }

  # Only attempt to put the file when the S3 Bucket (and related resources) are created
  depends_on = [
    aws_s3_bucket.streamlit_s3_bucket,
    aws_s3_bucket_notification.streamlit_s3_bucket,
    aws_s3_bucket_policy.streamlit_s3_bucket,
    aws_s3_bucket_versioning.streamlit_s3_bucket
  ]
}

[amarjandu]

Also effecting us....
I don't wanna use the workaround :(

Terraform v1.9.5
hashicorp/aws v5.67.0