aws_s3_bucket create fails with NoSuchBucket error listing resource tags #13008

camlow325 · 2020-04-24T20:42:04Z

Community Note

Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request
If you are interested in working on this issue or have submitted a pull request, please leave a comment

Terraform Version

$ terraform -v
Terraform v0.12.20
+ provider.aws v2.59.0

Affected Resource(s)

aws_s3_bucket

Terraform Configuration Files

resource "aws_s3_bucket" "test" {
  bucket        = "test"
  force_destroy = true

  acl = "private"

  server_side_encryption_configuration {
    rule {
      apply_server_side_encryption_by_default {
        sse_algorithm = "AES256"
      }
    }
  }

  tags = {
    Tag1 = "Tag1Value"
    Tag2 = "Tag2Value"
  }
}

Debug Output

Here are some snippets of the debug log around where the error occurs. The problem appears to occur when AWS returns a 404 Not Found error in response to the GetBucketTagging call for a new S3 bucket.

The debug log shows that the S3 bucket itself appears to be created properly:

2020-04-23T03:17:18.382Z [DEBUG] plugin.terraform-provider-aws_v2.54.0_x4: 2020/04/23 03:17:18 [DEBUG] [aws-sdk-go] DEBUG: Request s3/CreateBucket Details:
2020-04-23T03:17:18.382Z [DEBUG] plugin.terraform-provider-aws_v2.54.0_x4: ---[ REQUEST POST-SIGN ]-----------------------------
2020-04-23T03:17:18.382Z [DEBUG] plugin.terraform-provider-aws_v2.54.0_x4: PUT / HTTP/1.1
2020-04-23T03:17:18.382Z [DEBUG] plugin.terraform-provider-aws_v2.54.0_x4: Host: xxx.s3.us-west-2.amazonaws.com
2020-04-23T03:17:18.382Z [DEBUG] plugin.terraform-provider-aws_v2.54.0_x4: User-Agent: aws-sdk-go/1.29.24 (go1.13.7; linux; amd64) APN/1.0 HashiCorp/1.0 Terraform/0.12.20 (+https://www.terraform.io)
...
2020-04-23T03:17:18.382Z [DEBUG] plugin.terraform-provider-aws_v2.54.0_x4: 
2020-04-23T03:17:18.382Z [DEBUG] plugin.terraform-provider-aws_v2.54.0_x4: <CreateBucketConfiguration xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><LocationConstraint>us-west-2</LocationConstraint></CreateBucketConfiguration>
...
2020-04-23T03:17:19.210Z [DEBUG] plugin.terraform-provider-aws_v2.54.0_x4: 2020/04/23 03:17:19 [DEBUG] [aws-sdk-go] DEBUG: Response s3/CreateBucket Details:
2020-04-23T03:17:19.210Z [DEBUG] plugin.terraform-provider-aws_v2.54.0_x4: ---[ RESPONSE ]--------------------------------------
2020-04-23T03:17:19.210Z [DEBUG] plugin.terraform-provider-aws_v2.54.0_x4: HTTP/1.1 200 OK
2020-04-23T03:17:19.210Z [DEBUG] plugin.terraform-provider-aws_v2.54.0_x4: Connection: close
2020-04-23T03:17:19.210Z [DEBUG] plugin.terraform-provider-aws_v2.54.0_x4: Content-Length: 0
2020-04-23T03:17:19.210Z [DEBUG] plugin.terraform-provider-aws_v2.54.0_x4: Date: Thu, 23 Apr 2020 03:17:19 GMT
2020-04-23T03:17:19.210Z [DEBUG] plugin.terraform-provider-aws_v2.54.0_x4: Location: http://xxx.s3.amazonaws.com/
2020-04-23T03:17:19.210Z [DEBUG] plugin.terraform-provider-aws_v2.54.0_x4: Server: AmazonS3
...

When the error occurs, the subsequent GetBucketTagging request/response looks like this:

2020-04-23T03:17:19.210Z [DEBUG] plugin.terraform-provider-aws_v2.54.0_x4: 2020/04/23 03:17:19 [DEBUG] [aws-sdk-go] DEBUG: Request s3/GetBucketTagging Details:
2020-04-23T03:17:19.210Z [DEBUG] plugin.terraform-provider-aws_v2.54.0_x4: ---[ REQUEST POST-SIGN ]-----------------------------
2020-04-23T03:17:19.210Z [DEBUG] plugin.terraform-provider-aws_v2.54.0_x4: GET /?tagging= HTTP/1.1
2020-04-23T03:17:19.210Z [DEBUG] plugin.terraform-provider-aws_v2.54.0_x4: Host: xxx.s3.us-west-2.amazonaws.com
2020-04-23T03:17:19.210Z [DEBUG] plugin.terraform-provider-aws_v2.54.0_x4: User-Agent: aws-sdk-go/1.29.24 (go1.13.7; linux; amd64) APN/1.0 HashiCorp/1.0 Terraform/0.12.20 (+https://www.terraform.io)
...
2020-04-23T03:17:19.431Z [DEBUG] plugin.terraform-provider-aws_v2.54.0_x4: 2020/04/23 03:17:19 [DEBUG] [aws-sdk-go] DEBUG: Response s3/GetBucketTagging Details:
2020-04-23T03:17:19.431Z [DEBUG] plugin.terraform-provider-aws_v2.54.0_x4: ---[ RESPONSE ]--------------------------------------
2020-04-23T03:17:19.431Z [DEBUG] plugin.terraform-provider-aws_v2.54.0_x4: HTTP/1.1 404 Not Found
2020-04-23T03:17:19.431Z [DEBUG] plugin.terraform-provider-aws_v2.54.0_x4: Connection: close
2020-04-23T03:17:19.431Z [DEBUG] plugin.terraform-provider-aws_v2.54.0_x4: Transfer-Encoding: chunked
2020-04-23T03:17:19.431Z [DEBUG] plugin.terraform-provider-aws_v2.54.0_x4: Content-Type: application/xml
2020-04-23T03:17:19.431Z [DEBUG] plugin.terraform-provider-aws_v2.54.0_x4: Date: Thu, 23 Apr 2020 03:17:19 GMT
2020-04-23T03:17:19.431Z [DEBUG] plugin.terraform-provider-aws_v2.54.0_x4: Server: AmazonS3
...
2020-04-23T03:17:19.431Z [DEBUG] plugin.terraform-provider-aws_v2.54.0_x4: -----------------------------------------------------
2020-04-23T03:17:19.431Z [DEBUG] plugin.terraform-provider-aws_v2.54.0_x4: 2020/04/23 03:17:19 [DEBUG] [aws-sdk-go] <?xml version="1.0" encoding="UTF-8"?>
2020-04-23T03:17:19.431Z [DEBUG] plugin.terraform-provider-aws_v2.54.0_x4: <Error><Code>NoSuchBucket</Code><Message>The specified bucket does not exist</Message><BucketName>xxx</BucketName><RequestId>xxx</RequestId><HostId>xxx</HostId></Error>
2020-04-23T03:17:19.431Z [DEBUG] plugin.terraform-provider-aws_v2.54.0_x4: 2020/04/23 03:17:19 [DEBUG] [aws-sdk-go] DEBUG: Validate Response s3/GetBucketTagging failed, attempt 0/25, error NoSuchBucket: The specified bucket does not exist
2020-04-23T03:17:19.431Z [DEBUG] plugin.terraform-provider-aws_v2.54.0_x4: 	status code: 404, request id: xxx, host id: xxx
2020/04/23 03:17:19 [DEBUG] module.generic_data_pipeline.module.firehose.module.firehose_s3_bucket.aws_s3_bucket.primary[0]: apply errored, but we're indicating that via the Error pointer rather than returning it: error updating S3 Bucket (xxx) tags: error listing resource tags (xxx): NoSuchBucket: The specified bucket does not exist
	status code: 404, request id: xxx, host id: xxx

Panic Output

Expected Behavior

The S3 bucket is created properly and the terraform apply should succeed.

Actual Behavior

Most of the time, the expected behavior occurs. Intermittently, though, the S3 bucket is created successfully in AWS but the terraform apply fails when trying to retrieve tags for the bucket:

Error: error updating S3 Bucket (xxx) tags: error listing resource tags (xxx): NoSuchBucket: The specified bucket does not exist
	status code: 404, request id: xxx, host id: xxx

  on modules/common/s3/main.tf line 16, in resource "aws_s3_bucket" "test":
  16: resource "aws_s3_bucket" "test" {

If a second terraform apply is attempted after the error occurs, the terraform apply succeeds.

Given that the problem is intermittent and a later apply succeeds, it seems like the error may be due to an eventual consistency issue.

Steps to Reproduce

terraform apply

Important Factoids

References

This appears to be the same as the issue referenced in #12418, but this problem is still reproducible on the latest provider code (v2.59.0), which includes this fix.

The text was updated successfully, but these errors were encountered:

ewbankkit · 2020-04-24T20:55:01Z

Similar:

aws_s3_bucket_object eventual consistency error after object creation #12652
Error updating S3 Bucket tags: error setting resource tags: NoSuchBucket: The specified bucket does not exist #12907

lebedana · 2020-06-29T15:36:56Z

Same here.

$ terraform -v
Terraform v0.12.24
+ provider.aws v2.68.0

References: * hashicorp#13008 The AWS S3 service has eventual consistency considerations. If a GetBucketTagging call is made to obtain tags just after an S3 bucket is first created, AWS may return an HTTP 404 (NotFound) error with a NoSuchBucket error code. A fix was added for this in hashicorp#12418. It appears that the NoSuchBucket errors are not retried with this fix, however. This commit adds some extra logic which ensures that the code from the awserr.Error instance is evaluated for retry. Output for acceptance testing: ``` > make testacc TEST=./aws TESTARGS='-run=TestAccAWSS3Bucket_' ... --- PASS: TestAccAWSS3Bucket_shouldFailNotFound (22.93s) --- PASS: TestAccAWSS3Bucket_LifecycleRule_Expiration_EmptyConfigurationBlock (38.71s) --- PASS: TestAccAWSS3Bucket_forceDestroyWithEmptyPrefixes (40.46s) --- PASS: TestAccAWSS3Bucket_forceDestroy (40.47s) --- PASS: TestAccAWSS3Bucket_enableDefaultEncryption_whenAES256IsUsed (45.72s) --- PASS: TestAccAWSS3Bucket_basic (45.79s) --- PASS: TestAccAWSS3Bucket_forceDestroyWithObjectLockEnabled (47.59s) --- PASS: TestAccAWSS3Bucket_ReplicationExpectVersioningValidationError (51.85s) --- PASS: TestAccAWSS3Bucket_LifecycleBasic (95.95s) --- PASS: TestAccAWSS3Bucket_LifecycleExpireMarkerOnly (69.16s) --- PASS: TestAccAWSS3Bucket_enableDefaultEncryption_whenTypical (71.63s) --- PASS: TestAccAWSS3Bucket_objectLock (74.52s) --- PASS: TestAccAWSS3Bucket_disableDefaultEncryption_whenDefaultEncryptionIsEnabled (75.75s) --- PASS: TestAccAWSS3Bucket_region (42.47s) --- PASS: TestAccAWSS3Bucket_ReplicationWithoutPrefix (96.80s) --- PASS: TestAccAWSS3Bucket_WebsiteRoutingRules (77.87s) --- PASS: TestAccAWSS3Bucket_Versioning (106.51s) --- PASS: TestAccAWSS3Bucket_UpdateGrant (110.50s) --- PASS: TestAccAWSS3Bucket_GrantToAcl (71.89s) --- PASS: TestAccAWSS3Bucket_AclToGrant (72.06s) --- PASS: TestAccAWSS3Bucket_generatedName (47.42s) --- PASS: TestAccAWSS3Bucket_UpdateAcl (74.86s) --- PASS: TestAccAWSS3Bucket_namePrefix (44.58s) --- PASS: TestAccAWSS3Bucket_RequestPayer (75.62s) --- PASS: TestAccAWSS3Bucket_ReplicationWithoutStorageClass (97.00s) --- PASS: TestAccAWSS3Bucket_Cors_EmptyOrigin (47.36s) --- PASS: TestAccAWSS3Bucket_WebsiteRedirect (109.43s) --- PASS: TestAccAWSS3Bucket_Website_Simple (109.41s) --- PASS: TestAccAWSS3Bucket_ReplicationConfiguration_Rule_Destination_AddAccessControlTranslation (152.76s) --- PASS: TestAccAWSS3Bucket_acceleration (81.59s) --- PASS: TestAccAWSS3Bucket_Cors_Delete (36.58s) --- PASS: TestAccAWSS3Bucket_Bucket_EmptyString (44.94s) --- PASS: TestAccAWSS3Bucket_Logging (66.26s) --- PASS: TestAccAWSS3Bucket_Policy (104.79s) --- PASS: TestAccAWSS3Bucket_ReplicationConfiguration_Rule_Destination_AccessControlTranslation (176.75s) --- PASS: TestAccAWSS3Bucket_Cors_Update (71.71s) --- PASS: TestAccAWSS3Bucket_tagsWithNoSystemTags (135.94s) --- PASS: TestAccAWSS3Bucket_tagsWithSystemTags (168.39s) --- PASS: TestAccAWSS3Bucket_Replication (270.65s) --- PASS: TestAccAWSS3Bucket_ReplicationSchemaV2 (273.25s) ```

github-actions · 2021-09-02T22:27:51Z

This functionality has been released in v3.57.0 of the Terraform AWS Provider. Please see the Terraform documentation on provider versioning or reach out if you need any assistance upgrading.

For further feature requests or bug reports with this functionality, please create a new GitHub issue following the template. Thank you!

github-actions · 2022-06-10T02:34:36Z

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.

ghost added the service/s3 Issues and PRs that pertain to the s3 service. label Apr 24, 2020

github-actions bot added the needs-triage Waiting for first response or review from a maintainer. label Apr 24, 2020

camlow325 mentioned this issue Apr 24, 2020

resource/aws_s3_bucket: Retry on GetBucketTagging 404 Errors #13009

Merged

bflad added bug Addresses a defect in current functionality. and removed needs-triage Waiting for first response or review from a maintainer. labels Apr 28, 2020

ewbankkit closed this as completed in #13009 Aug 30, 2021

github-actions bot added this to the v3.57.0 milestone Aug 30, 2021

github-actions bot locked as resolved and limited conversation to collaborators Jun 10, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

aws_s3_bucket create fails with NoSuchBucket error listing resource tags #13008

aws_s3_bucket create fails with NoSuchBucket error listing resource tags #13008

camlow325 commented Apr 24, 2020 •

edited

Loading

ewbankkit commented Apr 24, 2020

lebedana commented Jun 29, 2020

github-actions bot commented Sep 2, 2021

github-actions bot commented Jun 10, 2022

aws_s3_bucket create fails with NoSuchBucket error listing resource tags #13008

aws_s3_bucket create fails with NoSuchBucket error listing resource tags #13008

Comments

camlow325 commented Apr 24, 2020 • edited Loading

Community Note

Terraform Version

Affected Resource(s)

Terraform Configuration Files

Debug Output

Panic Output

Expected Behavior

Actual Behavior

Steps to Reproduce

Important Factoids

References

ewbankkit commented Apr 24, 2020

lebedana commented Jun 29, 2020

github-actions bot commented Sep 2, 2021

github-actions bot commented Jun 10, 2022

camlow325 commented Apr 24, 2020 •

edited

Loading