Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Generate S3 presigned url with S3 Object Lambda Access Point not working #3678

Open
sakurai-ryo opened this issue Apr 21, 2023 · 2 comments · May be fixed by #3688
Open

Generate S3 presigned url with S3 Object Lambda Access Point not working #3678

sakurai-ryo opened this issue Apr 21, 2023 · 2 comments · May be fixed by #3688
Labels
documentation This is a problem with documentation. needs-review p2 This is a standard priority issue s3

Comments

@sakurai-ryo
Copy link

Describe the bug

Hi Team.
I am using S3 Object Lambdan Access Point to execute boto3 generate_presigned_url method.
But when I access the generated URL, I get a MissingAuthenticationToken error.

If I use the @aws-sdk/s3-request-presigner module to generate it with JavaScript, I can access it via URL.

Expected Behavior

Being able to access S3 objects via a presigned URL.

Current Behavior

I got the following error in xml format

<Error>
  <Code>MissingAuthenticationToken</Code>
  <Message>Missing authentication token.</Message>
  <RequestId>a5b8fde1-2d48-4093-98e0-7883a955a1c6</RequestId>
  <HostId>{host-id}</HostId>
</Error>

Reproduction Steps

The minimal code to reproduce is below.

import boto3
import logging

boto3.set_stream_logger('', logging.DEBUG)


if __name__ == "__main__":
    sess = boto3.Session(profile_name="xxxxxx")
    s3_client = sess.client("s3")

    s3_object_lambda_access_point = "your object lambda access point arn"
    key = "your object key"

    url = s3_client.generate_presigned_url(
        ClientMethod="get_object",
        Params={"Bucket": s3_object_lambda_access_point, "Key": key},
        ExpiresIn=60,
    )
    print(url)

Possible Solution

No response

Additional Information/Context

Debug log for boto3 is below

% python presign.py
2023-04-21 12:31:57,908 botocore.hooks [DEBUG] Changing event name from creating-client-class.iot-data to creating-client-class.iot-data-plane
2023-04-21 12:31:57,909 botocore.hooks [DEBUG] Changing event name from before-call.apigateway to before-call.api-gateway
2023-04-21 12:31:57,909 botocore.hooks [DEBUG] Changing event name from request-created.machinelearning.Predict to request-created.machine-learning.Predict
2023-04-21 12:31:57,910 botocore.hooks [DEBUG] Changing event name from before-parameter-build.autoscaling.CreateLaunchConfiguration to before-parameter-build.auto-scaling.CreateLaunchConfiguration
2023-04-21 12:31:57,910 botocore.hooks [DEBUG] Changing event name from before-parameter-build.route53 to before-parameter-build.route-53
2023-04-21 12:31:57,910 botocore.hooks [DEBUG] Changing event name from request-created.cloudsearchdomain.Search to request-created.cloudsearch-domain.Search
2023-04-21 12:31:57,911 botocore.hooks [DEBUG] Changing event name from docs.*.autoscaling.CreateLaunchConfiguration.complete-section to docs.*.auto-scaling.CreateLaunchConfiguration.complete-section
2023-04-21 12:31:57,912 botocore.hooks [DEBUG] Changing event name from before-parameter-build.logs.CreateExportTask to before-parameter-build.cloudwatch-logs.CreateExportTask
2023-04-21 12:31:57,912 botocore.hooks [DEBUG] Changing event name from docs.*.logs.CreateExportTask.complete-section to docs.*.cloudwatch-logs.CreateExportTask.complete-section
2023-04-21 12:31:57,912 botocore.hooks [DEBUG] Changing event name from before-parameter-build.cloudsearchdomain.Search to before-parameter-build.cloudsearch-domain.Search
2023-04-21 12:31:57,912 botocore.hooks [DEBUG] Changing event name from docs.*.cloudsearchdomain.Search.complete-section to docs.*.cloudsearch-domain.Search.complete-section
2023-04-21 12:31:57,912 botocore.session [DEBUG] Setting config variable for profile to 'xxxxxxxxx'
2023-04-21 12:31:57,915 botocore.utils [DEBUG] IMDS ENDPOINT: http://169.254.169.254/
2023-04-21 12:31:57,920 botocore.credentials [DEBUG] Skipping environment variable credential check because profile name was explicitly set.
2023-04-21 12:31:57,920 botocore.credentials [DEBUG] Looking for credentials via: assume-role
2023-04-21 12:31:57,920 botocore.credentials [DEBUG] Looking for credentials via: assume-role-with-web-identity
2023-04-21 12:31:57,920 botocore.credentials [DEBUG] Looking for credentials via: sso
2023-04-21 12:31:57,920 botocore.credentials [DEBUG] Looking for credentials via: shared-credentials-file
2023-04-21 12:31:57,920 botocore.credentials [INFO] Found credentials in shared credentials file: ~/.aws/credentials
2023-04-21 12:31:57,921 botocore.loaders [DEBUG] Loading JSON file: /Users/user/.anyenv/envs/pyenv/versions/3.11.0/lib/python3.11/site-packages/botocore/data/endpoints.json
2023-04-21 12:31:57,927 botocore.loaders [DEBUG] Loading JSON file: /Users/user/.anyenv/envs/pyenv/versions/3.11.0/lib/python3.11/site-packages/botocore/data/sdk-default-configuration.json
2023-04-21 12:31:57,927 botocore.hooks [DEBUG] Event choose-service-name: calling handler <function handle_service_name_alias at 0x1048f39c0>
2023-04-21 12:31:57,937 botocore.loaders [DEBUG] Loading JSON file: /Users/user/.anyenv/envs/pyenv/versions/3.11.0/lib/python3.11/site-packages/botocore/data/s3/2006-03-01/service-2.json
2023-04-21 12:31:57,951 botocore.loaders [DEBUG] Loading JSON file: /Users/user/.anyenv/envs/pyenv/versions/3.11.0/lib/python3.11/site-packages/botocore/data/s3/2006-03-01/endpoint-rule-set-1.json.gz
2023-04-21 12:31:57,954 botocore.loaders [DEBUG] Loading JSON file: /Users/user/.anyenv/envs/pyenv/versions/3.11.0/lib/python3.11/site-packages/botocore/data/partitions.json
2023-04-21 12:31:57,955 botocore.hooks [DEBUG] Event creating-client-class.s3: calling handler <function add_generate_presigned_post at 0x104829b20>
2023-04-21 12:31:57,955 botocore.hooks [DEBUG] Event creating-client-class.s3: calling handler <function lazy_call.<locals>._handler at 0x103739760>
2023-04-21 12:31:57,972 botocore.hooks [DEBUG] Event creating-client-class.s3: calling handler <function add_generate_presigned_url at 0x1048298a0>
2023-04-21 12:31:57,992 botocore.endpoint [DEBUG] Setting s3 timeout as (60, 60)
2023-04-21 12:31:57,994 botocore.loaders [DEBUG] Loading JSON file: /Users/user/.anyenv/envs/pyenv/versions/3.11.0/lib/python3.11/site-packages/botocore/data/_retry.json
2023-04-21 12:31:58,003 botocore.client [DEBUG] Registering retry handlers for service: s3
2023-04-21 12:31:58,003 botocore.utils [DEBUG] Registering S3 region redirector handler
2023-04-21 12:31:58,004 botocore.hooks [DEBUG] Event before-endpoint-resolution.s3: calling handler <function customize_endpoint_resolver_builtins at 0x10491f7e0>
2023-04-21 12:31:58,004 botocore.hooks [DEBUG] Event before-endpoint-resolution.s3: calling handler <bound method S3RegionRedirectorv2.redirect_from_cache of <botocore.utils.S3RegionRedirectorv2 object at 0x109d08f50>>
2023-04-21 12:31:58,004 botocore.regions [DEBUG] Calling endpoint provider with parameters: {'Bucket': 'arn:aws:s3-object-lambda:ap-northeast-1:xxxxxxxxxxxxxx:accesspoint/test', 'Region': 'ap-northeast-1', 'UseFIPS': False, 'UseDualStack': False, 'ForcePathStyle': False, 'Accelerate': False, 'UseGlobalEndpoint': False, 'DisableMultiRegionAccessPoints': False, 'UseArnRegion': True}
2023-04-21 12:31:58,004 botocore.regions [DEBUG] Endpoint provider result: https://xxxxxxxxxxx.s3-object-lambda.ap-northeast-1.amazonaws.com
2023-04-21 12:31:58,004 botocore.regions [DEBUG] Selecting from endpoint provider's list of auth schemes: "sigv4". User selected auth scheme is: "None"
2023-04-21 12:31:58,004 botocore.regions [DEBUG] Selected auth type "v4" as "v4" with signing context params: {'region': 'ap-northeast-1', 'signing_name': 's3-object-lambda', 'disableDoubleEncoding': True}
2023-04-21 12:31:58,004 botocore.hooks [DEBUG] Event before-parameter-build.s3.GetObject: calling handler <function sse_md5 at 0x10491d580>
2023-04-21 12:31:58,004 botocore.hooks [DEBUG] Event before-parameter-build.s3.GetObject: calling handler <function validate_bucket_name at 0x10491d4e0>
2023-04-21 12:31:58,004 botocore.hooks [DEBUG] Event before-parameter-build.s3.GetObject: calling handler <function remove_bucket_from_url_paths_from_model at 0x10491f600>
2023-04-21 12:31:58,004 botocore.hooks [DEBUG] Event before-parameter-build.s3.GetObject: calling handler <bound method S3RegionRedirectorv2.annotate_request_context of <botocore.utils.S3RegionRedirectorv2 object at 0x109d08f50>>
2023-04-21 12:31:58,004 botocore.hooks [DEBUG] Event before-parameter-build.s3.GetObject: calling handler <function generate_idempotent_uuid at 0x10491d300>
2023-04-21 12:31:58,005 botocore.hooks [DEBUG] Event choose-signer.s3.GetObject: calling handler <bound method ClientCreator._default_s3_presign_to_sigv2 of <botocore.client.ClientCreator object at 0x10982f210>>
2023-04-21 12:31:58,005 botocore.hooks [DEBUG] Event before-sign.s3.GetObject: calling handler <function remove_arn_from_signing_path at 0x10491f740>
2023-04-21 12:31:58,005 botocore.auth [DEBUG] Calculating signature using hmacv1 auth.
2023-04-21 12:31:58,005 botocore.auth [DEBUG] HTTP request method: GET
2023-04-21 12:31:58,005 botocore.auth [DEBUG] StringToSign:

SDK version used

1.26.117

Environment details (OS name and version, etc.)

MacOS Ventura
@sakurai-ryo sakurai-ryo added bug This issue is a confirmed bug. needs-triage This issue or PR still needs to be triaged. labels Apr 21, 2023
@tim-finnigan tim-finnigan self-assigned this Apr 21, 2023
@tim-finnigan
Copy link
Contributor

Hi @sakurai-ryo thanks for reaching out. Have you tried using SigV4 as suggested in this re:Post answer? For example:
s3 = boto3.client('s3', config=Config(signature_version='s3v4'))

@tim-finnigan tim-finnigan added response-requested Waiting on additional information or feedback. s3 and removed bug This issue is a confirmed bug. needs-triage This issue or PR still needs to be triaged. labels Apr 21, 2023
@sakurai-ryo
Copy link
Author

@tim-finnigan
It worked correctly, thank you!
I will send a PR to add this to the documentation.
https://github.com/boto/boto3/blob/develop/docs/source/guide/s3-presigned-urls.rst

@github-actions github-actions bot removed the response-requested Waiting on additional information or feedback. label Apr 24, 2023
@sakurai-ryo sakurai-ryo linked a pull request May 2, 2023 that will close this issue
Open
@tim-finnigan tim-finnigan removed their assignment May 10, 2023
@tim-finnigan tim-finnigan added documentation This is a problem with documentation. needs-review p2 This is a standard priority issue labels May 10, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
documentation This is a problem with documentation. needs-review p2 This is a standard priority issue s3
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

docs: Added description of Sigv4 when generating PreSigned url sakurai-ryo/boto3
2 participants
@sakurai-ryo @tim-finnigan