Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Can't use optFn to set region for S3 Express operations (credentials issue) #2777

Open
2 tasks done
stefansundin opened this issue Sep 6, 2024 · 1 comment
Open
2 tasks done

Can't use optFn to set region for S3 Express operations (credentials issue) #2777

stefansundin opened this issue Sep 6, 2024 · 1 comment
Labels
bug This issue is a bug. p2 This is a standard priority issue queued This issues is on the AWS team's backlog workaround-available

Comments

@stefansundin
Copy link

Acknowledgements

Describe the bug

S3 Express buckets require credentials that are implicitly retrieved with the CreateSession API operation. However, if the region is set using an optFn, that region is not propagated to the CreateSession call, and results in this error:

operation error S3: GetObject, get identity: get credentials: operation error S3: CreateSession, https response error StatusCode: 0, RequestID: , HostID: , request send failed, Get "https://my-test-bucket--usw2-az1--x-s3.s3express-usw2-az1.us-west-1.amazonaws.com/?session=": dial tcp: lookup my-test-bucket--usw2-az1--x-s3.s3express-usw2-az1.us-west-1.amazonaws.com: no such host

In this case my S3 client was initialized with the us-west-1 region, but I'm attempting to set the region to us-west-2 in an optFn, but this is not respected in the CreateSession call.

Expected Behavior

I expect optFn functions to be forwarded to the CreateSession operation.

Current Behavior

optFn functions do not affect S3 Express credentials fetching.

Reproduction Steps

package main

import (
	"context"
	"fmt"
	"os"

	"github.com/aws/aws-sdk-go-v2/aws"
	"github.com/aws/aws-sdk-go-v2/config"
	"github.com/aws/aws-sdk-go-v2/service/s3"
)

func main() {
	bucket := "my-test-bucket--usw2-az1--x-s3" // this S3 Express bucket is in us-west-2
	key := "test"
	initialRegion := "us-west-1" // incorrect region for my S3 Express bucket
	debug := true

	// I want to set the correct region using an optFn when making the GetObject call:
	setRegionFn := func(o *s3.Options) {
		o.Region = "us-west-2"
	}

	// Initialize the AWS SDK
	cfg, err := config.LoadDefaultConfig(
		context.TODO(),
		config.WithRegion(initialRegion),
		func(o *config.LoadOptions) error {
			if debug {
				var lm aws.ClientLogMode = aws.LogRequest | aws.LogResponse
				o.ClientLogMode = &lm
			}
			return nil
		},
	)
	if err != nil {
		fmt.Fprintln(os.Stderr, err)
		os.Exit(1)
	}

	client := s3.NewFromConfig(cfg)

	obj, err := client.GetObject(context.TODO(), &s3.GetObjectInput{
		Bucket: aws.String(bucket),
		Key:    aws.String(key),
	}, setRegionFn) // <-- optFn used here

	if err != nil {
		fmt.Fprintln(os.Stderr, err)
		os.Exit(1)
	}

	fmt.Fprintf(os.Stderr, "GetObject ContentLength: %d\n", aws.ToInt64(obj.ContentLength))
}

Possible Solution

Apply optFn functions first, and then clone the resulting client options before calling CreateSession?

Additional Information/Context

No response

AWS Go SDK V2 Module Versions Used

example github.com/aws/aws-sdk-go-v2@v1.30.5
example github.com/aws/aws-sdk-go-v2/aws/protocol/eventstream@v1.6.4
example github.com/aws/aws-sdk-go-v2/config@v1.27.33
example github.com/aws/aws-sdk-go-v2/credentials@v1.17.32
example github.com/aws/aws-sdk-go-v2/feature/ec2/imds@v1.16.13
example github.com/aws/aws-sdk-go-v2/internal/configsources@v1.3.17
example github.com/aws/aws-sdk-go-v2/internal/endpoints/v2@v2.6.17
example github.com/aws/aws-sdk-go-v2/internal/ini@v1.8.1
example github.com/aws/aws-sdk-go-v2/internal/v4a@v1.3.17
example github.com/aws/aws-sdk-go-v2/service/internal/accept-encoding@v1.11.4
example github.com/aws/aws-sdk-go-v2/service/internal/checksum@v1.3.19
example github.com/aws/aws-sdk-go-v2/service/internal/presigned-url@v1.11.19
example github.com/aws/aws-sdk-go-v2/service/internal/s3shared@v1.17.17
example github.com/aws/aws-sdk-go-v2/service/s3@v1.61.2
example github.com/aws/aws-sdk-go-v2/service/sso@v1.22.7
example github.com/aws/aws-sdk-go-v2/service/ssooidc@v1.26.7
example github.com/aws/aws-sdk-go-v2/service/sts@v1.30.7
example github.com/aws/smithy-go@v1.20.4
example go@1.23.1
github.com/aws/aws-sdk-go-v2@v1.30.5 github.com/aws/smithy-go@v1.20.4
github.com/aws/aws-sdk-go-v2@v1.30.5 github.com/jmespath/go-jmespath@v0.4.0
github.com/aws/aws-sdk-go-v2@v1.30.5 go@1.21
github.com/aws/aws-sdk-go-v2/aws/protocol/eventstream@v1.6.4 github.com/aws/smithy-go@v1.20.4
github.com/aws/aws-sdk-go-v2/aws/protocol/eventstream@v1.6.4 go@1.21
github.com/aws/aws-sdk-go-v2/config@v1.27.33 github.com/aws/aws-sdk-go-v2@v1.30.5
github.com/aws/aws-sdk-go-v2/config@v1.27.33 github.com/aws/aws-sdk-go-v2/credentials@v1.17.32
github.com/aws/aws-sdk-go-v2/config@v1.27.33 github.com/aws/aws-sdk-go-v2/feature/ec2/imds@v1.16.13
github.com/aws/aws-sdk-go-v2/config@v1.27.33 github.com/aws/aws-sdk-go-v2/internal/ini@v1.8.1
github.com/aws/aws-sdk-go-v2/config@v1.27.33 github.com/aws/aws-sdk-go-v2/service/sso@v1.22.7
github.com/aws/aws-sdk-go-v2/config@v1.27.33 github.com/aws/aws-sdk-go-v2/service/ssooidc@v1.26.7
github.com/aws/aws-sdk-go-v2/config@v1.27.33 github.com/aws/aws-sdk-go-v2/service/sts@v1.30.7
github.com/aws/aws-sdk-go-v2/config@v1.27.33 github.com/aws/smithy-go@v1.20.4
github.com/aws/aws-sdk-go-v2/config@v1.27.33 github.com/aws/aws-sdk-go-v2/internal/configsources@v1.3.17
github.com/aws/aws-sdk-go-v2/config@v1.27.33 github.com/aws/aws-sdk-go-v2/internal/endpoints/v2@v2.6.17
github.com/aws/aws-sdk-go-v2/config@v1.27.33 github.com/aws/aws-sdk-go-v2/service/internal/accept-encoding@v1.11.4
github.com/aws/aws-sdk-go-v2/config@v1.27.33 github.com/aws/aws-sdk-go-v2/service/internal/presigned-url@v1.11.19
github.com/aws/aws-sdk-go-v2/config@v1.27.33 go@1.21
github.com/aws/aws-sdk-go-v2/credentials@v1.17.32 github.com/aws/aws-sdk-go-v2@v1.30.5
github.com/aws/aws-sdk-go-v2/credentials@v1.17.32 github.com/aws/aws-sdk-go-v2/feature/ec2/imds@v1.16.13
github.com/aws/aws-sdk-go-v2/credentials@v1.17.32 github.com/aws/aws-sdk-go-v2/service/sso@v1.22.7
github.com/aws/aws-sdk-go-v2/credentials@v1.17.32 github.com/aws/aws-sdk-go-v2/service/ssooidc@v1.26.7
github.com/aws/aws-sdk-go-v2/credentials@v1.17.32 github.com/aws/aws-sdk-go-v2/service/sts@v1.30.7
github.com/aws/aws-sdk-go-v2/credentials@v1.17.32 github.com/aws/smithy-go@v1.20.4
github.com/aws/aws-sdk-go-v2/credentials@v1.17.32 github.com/aws/aws-sdk-go-v2/internal/configsources@v1.3.17
github.com/aws/aws-sdk-go-v2/credentials@v1.17.32 github.com/aws/aws-sdk-go-v2/internal/endpoints/v2@v2.6.17
github.com/aws/aws-sdk-go-v2/credentials@v1.17.32 github.com/aws/aws-sdk-go-v2/service/internal/accept-encoding@v1.11.4
github.com/aws/aws-sdk-go-v2/credentials@v1.17.32 github.com/aws/aws-sdk-go-v2/service/internal/presigned-url@v1.11.19
github.com/aws/aws-sdk-go-v2/credentials@v1.17.32 go@1.21
github.com/aws/aws-sdk-go-v2/feature/ec2/imds@v1.16.13 github.com/aws/aws-sdk-go-v2@v1.30.5
github.com/aws/aws-sdk-go-v2/feature/ec2/imds@v1.16.13 github.com/aws/smithy-go@v1.20.4
github.com/aws/aws-sdk-go-v2/feature/ec2/imds@v1.16.13 go@1.21
github.com/aws/aws-sdk-go-v2/internal/configsources@v1.3.17 github.com/aws/aws-sdk-go-v2@v1.30.5
github.com/aws/aws-sdk-go-v2/internal/configsources@v1.3.17 github.com/aws/smithy-go@v1.20.4
github.com/aws/aws-sdk-go-v2/internal/configsources@v1.3.17 go@1.21
github.com/aws/aws-sdk-go-v2/internal/endpoints/v2@v2.6.17 github.com/aws/aws-sdk-go-v2@v1.30.5
github.com/aws/aws-sdk-go-v2/internal/endpoints/v2@v2.6.17 github.com/aws/smithy-go@v1.20.4
github.com/aws/aws-sdk-go-v2/internal/endpoints/v2@v2.6.17 go@1.21
github.com/aws/aws-sdk-go-v2/internal/ini@v1.8.1 go@1.21
github.com/aws/aws-sdk-go-v2/internal/v4a@v1.3.17 github.com/aws/aws-sdk-go-v2@v1.30.5
github.com/aws/aws-sdk-go-v2/internal/v4a@v1.3.17 github.com/aws/smithy-go@v1.20.4
github.com/aws/aws-sdk-go-v2/internal/v4a@v1.3.17 go@1.21
github.com/aws/aws-sdk-go-v2/service/internal/accept-encoding@v1.11.4 github.com/aws/smithy-go@v1.20.4
github.com/aws/aws-sdk-go-v2/service/internal/accept-encoding@v1.11.4 go@1.21
github.com/aws/aws-sdk-go-v2/service/internal/checksum@v1.3.19 github.com/aws/aws-sdk-go-v2@v1.30.5
github.com/aws/aws-sdk-go-v2/service/internal/checksum@v1.3.19 github.com/aws/aws-sdk-go-v2/service/internal/presigned-url@v1.11.19
github.com/aws/aws-sdk-go-v2/service/internal/checksum@v1.3.19 github.com/aws/smithy-go@v1.20.4
github.com/aws/aws-sdk-go-v2/service/internal/checksum@v1.3.19 go@1.21
github.com/aws/aws-sdk-go-v2/service/internal/presigned-url@v1.11.19 github.com/aws/aws-sdk-go-v2@v1.30.5
github.com/aws/aws-sdk-go-v2/service/internal/presigned-url@v1.11.19 github.com/aws/smithy-go@v1.20.4
github.com/aws/aws-sdk-go-v2/service/internal/presigned-url@v1.11.19 go@1.21
github.com/aws/aws-sdk-go-v2/service/internal/s3shared@v1.17.17 github.com/aws/aws-sdk-go-v2@v1.30.5
github.com/aws/aws-sdk-go-v2/service/internal/s3shared@v1.17.17 github.com/aws/smithy-go@v1.20.4
github.com/aws/aws-sdk-go-v2/service/internal/s3shared@v1.17.17 go@1.21
github.com/aws/aws-sdk-go-v2/service/s3@v1.61.2 github.com/aws/aws-sdk-go-v2@v1.30.5
github.com/aws/aws-sdk-go-v2/service/s3@v1.61.2 github.com/aws/aws-sdk-go-v2/aws/protocol/eventstream@v1.6.4
github.com/aws/aws-sdk-go-v2/service/s3@v1.61.2 github.com/aws/aws-sdk-go-v2/internal/configsources@v1.3.17
github.com/aws/aws-sdk-go-v2/service/s3@v1.61.2 github.com/aws/aws-sdk-go-v2/internal/endpoints/v2@v2.6.17
github.com/aws/aws-sdk-go-v2/service/s3@v1.61.2 github.com/aws/aws-sdk-go-v2/internal/v4a@v1.3.17
github.com/aws/aws-sdk-go-v2/service/s3@v1.61.2 github.com/aws/aws-sdk-go-v2/service/internal/accept-encoding@v1.11.4
github.com/aws/aws-sdk-go-v2/service/s3@v1.61.2 github.com/aws/aws-sdk-go-v2/service/internal/checksum@v1.3.19
github.com/aws/aws-sdk-go-v2/service/s3@v1.61.2 github.com/aws/aws-sdk-go-v2/service/internal/presigned-url@v1.11.19
github.com/aws/aws-sdk-go-v2/service/s3@v1.61.2 github.com/aws/aws-sdk-go-v2/service/internal/s3shared@v1.17.17
github.com/aws/aws-sdk-go-v2/service/s3@v1.61.2 github.com/aws/smithy-go@v1.20.4
github.com/aws/aws-sdk-go-v2/service/s3@v1.61.2 go@1.21
github.com/aws/aws-sdk-go-v2/service/sso@v1.22.7 github.com/aws/aws-sdk-go-v2@v1.30.5
github.com/aws/aws-sdk-go-v2/service/sso@v1.22.7 github.com/aws/aws-sdk-go-v2/internal/configsources@v1.3.17
github.com/aws/aws-sdk-go-v2/service/sso@v1.22.7 github.com/aws/aws-sdk-go-v2/internal/endpoints/v2@v2.6.17
github.com/aws/aws-sdk-go-v2/service/sso@v1.22.7 github.com/aws/smithy-go@v1.20.4
github.com/aws/aws-sdk-go-v2/service/sso@v1.22.7 go@1.21
github.com/aws/aws-sdk-go-v2/service/ssooidc@v1.26.7 github.com/aws/aws-sdk-go-v2@v1.30.5
github.com/aws/aws-sdk-go-v2/service/ssooidc@v1.26.7 github.com/aws/aws-sdk-go-v2/internal/configsources@v1.3.17
github.com/aws/aws-sdk-go-v2/service/ssooidc@v1.26.7 github.com/aws/aws-sdk-go-v2/internal/endpoints/v2@v2.6.17
github.com/aws/aws-sdk-go-v2/service/ssooidc@v1.26.7 github.com/aws/smithy-go@v1.20.4
github.com/aws/aws-sdk-go-v2/service/ssooidc@v1.26.7 go@1.21
github.com/aws/aws-sdk-go-v2/service/sts@v1.30.7 github.com/aws/aws-sdk-go-v2@v1.30.5
github.com/aws/aws-sdk-go-v2/service/sts@v1.30.7 github.com/aws/aws-sdk-go-v2/internal/configsources@v1.3.17
github.com/aws/aws-sdk-go-v2/service/sts@v1.30.7 github.com/aws/aws-sdk-go-v2/internal/endpoints/v2@v2.6.17
github.com/aws/aws-sdk-go-v2/service/sts@v1.30.7 github.com/aws/aws-sdk-go-v2/service/internal/accept-encoding@v1.11.4
github.com/aws/aws-sdk-go-v2/service/sts@v1.30.7 github.com/aws/aws-sdk-go-v2/service/internal/presigned-url@v1.11.19
github.com/aws/aws-sdk-go-v2/service/sts@v1.30.7 github.com/aws/smithy-go@v1.20.4
github.com/aws/aws-sdk-go-v2/service/sts@v1.30.7 go@1.21
github.com/aws/smithy-go@v1.20.4 go@1.21
go@1.23.1 toolchain@go1.23.1

Compiler and Version used

go version go1.23.1 darwin/arm64

Operating System and version

macOS 14.6.1
@stefansundin stefansundin added bug This issue is a bug. needs-triage This issue or PR still needs to be triaged. labels Sep 6, 2024
@lucix-aws lucix-aws added p2 This is a standard priority issue queued This issues is on the AWS team's backlog and removed needs-triage This issue or PR still needs to be triaged. labels Sep 6, 2024
@lucix-aws
Copy link
Contributor

lucix-aws commented Sep 6, 2024 •
edited
Loading

Agreed. Workaround for the time being is to have multiple clients per region / config.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug This issue is a bug. p2 This is a standard priority issue queued This issues is on the AWS team's backlog workaround-available
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@stefansundin @lucix-aws