Skip to content
This repository has been archived by the owner on Apr 26, 2024. It is now read-only.

push_to_s3 and pull_from_s3 do not support additional AwsCredentials block parameters #313

Closed
1 task done
markbruning opened this issue Sep 18, 2023 · 0 comments · Fixed by #322
Closed
1 task done

Comments

@markbruning
Copy link
Contributor

Expectation / Proposal

deployments.steps:push_to_s3 and deployments.steps:pull_from_s3 do not support certain properties on the AwsCredentials block (profile_name, aws_client_parameters), causing the call to boto3.client to fail with bad parameters.

The processing of credentials and client_parameters in these functions does not match the signature of the resolved block input from a loaded AwsCredentials block. When **inputs is passed to Call.new in prefect\deployments\steps\core.py the block is resolved as with the signature

{
    'aws_access_key_id': '<key>',
    'aws_secret_access_key': '<secret>',
    'aws_session_token': None,
    'profile_name': '<profile>',
    'region_name': None,
    'aws_client_parameters': {
        'api_version': None,
        'use_ssl': True,
        'verify': True,
        'verify_cert_path': None,
        'endpoint_url': '<endpoint>',
        'config': None
    }
}

And then things like profile_name and aws_client_parameters are not recognized as kwargs for the boto3.client command.

I propose a function to process the AwsCredentials block signature to something that boto3.client supports.

Traceback / Example

Traceback (most recent call last):
  File "<repo>\.venv\prefect\lib\site-packages\prefect\deployments\steps\core.py", line 124, in run_steps
    step_output = await run_step(step, upstream_outputs)   
  File "<repo>\.venv\prefect\lib\site-packages\prefect\deployments\steps\core.py", line 95, in run_step
    result = await from_async.call_soon_in_new_thread(     
  File "<repo>\.venv\prefect\lib\site-packages\prefect\_internal\concurrency\calls.py", line 291, in aresult
    return await asyncio.wrap_future(self.future)
  File "<repo>\.venv\prefect\lib\site-packages\prefect\_internal\concurrency\calls.py", line 315, in _run_sync
    result = self.fn(*self.args, **self.kwargs)
  File "<repo>\.venv\prefect\lib\site-packages\prefect_aws\deployments\steps.py", line 99, in push_to_s3
    client = boto3.client(
  File "<repo>\.venv\prefect\lib\site-packages\boto3\__init__.py", line 92, in client
    return _get_default_session().client(*args, **kwargs)  
TypeError: Session.client() got an unexpected keyword argument 'aws_client_parameters'
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant