-
Notifications
You must be signed in to change notification settings - Fork 1.6k
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Migrate
prefect-aws
to core (#12888)
- Loading branch information
Showing
55 changed files
with
14,820 additions
and
6 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,6 +1,6 @@ | ||
collectionName: prefect-aws | ||
author: Prefect | ||
authorUrl: https://prefect.io | ||
documentation: https://prefecthq.github.io/prefect-aws/ | ||
documentation: prefect-aws | ||
iconUrl: /img/collections/aws.png | ||
tag: AWS |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,6 @@ | ||
--- | ||
description: Tasks for interacting with AWS Batch | ||
notes: This documentation page is generated from source file docstrings. | ||
--- | ||
|
||
::: prefect_aws.batch |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,6 @@ | ||
--- | ||
description: Task for waiting on a long-running AWS job | ||
notes: This documentation page is generated from source file docstrings. | ||
--- | ||
|
||
::: prefect_aws.client_waiter |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,6 @@ | ||
--- | ||
description: Module handling AWS credentials | ||
notes: This documentation page is generated from source file docstrings. | ||
--- | ||
|
||
::: prefect_aws.credentials |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,6 @@ | ||
--- | ||
description: Prefect deployment steps for managing deployment code storage via AWS S3. | ||
notes: This documentation page is generated from source file docstrings. | ||
--- | ||
|
||
::: prefect_aws.deployments.steps |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,6 @@ | ||
--- | ||
description: Integrations with the Amazon Elastic Container Service. | ||
notes: This documentation page is generated from source file docstrings. | ||
--- | ||
|
||
::: prefect_aws.ecs |
Large diffs are not rendered by default.
Oops, something went wrong.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,6 @@ | ||
--- | ||
description: Worker integration with the Amazon Elastic Container Service. | ||
notes: This documentation page is generated from source file docstrings. | ||
--- | ||
|
||
::: prefect_aws.workers.ecs_worker |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,6 @@ | ||
--- | ||
description: Tasks for interacting with AWS Glue Job | ||
notes: This documentation page is generated from source file docstrings. | ||
--- | ||
|
||
::: prefect_aws.glue_job |
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file not shown.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,135 @@ | ||
# `prefect-aws` | ||
|
||
<p align="center"> | ||
<a href="https://pypi.python.org/pypi/prefect-aws/" alt="PyPI version"> | ||
<img alt="PyPI" src="https://img.shields.io/pypi/v/prefect-aws?color=26272B&labelColor=090422"></a> | ||
<a href="https://pepy.tech/badge/prefect-aws/" alt="Downloads"> | ||
<img src="https://img.shields.io/pypi/dm/prefect-aws?color=26272B&labelColor=090422" /></a> | ||
</p> | ||
|
||
## Welcome | ||
|
||
`prefect-aws` makes it easy to leverage the capabilities of AWS in your workflows. | ||
|
||
## Getting started | ||
|
||
### Installation | ||
|
||
Prefect requires Python 3.8 or newer. | ||
|
||
We recommend using a Python virtual environment manager such as pipenv, conda, or virtualenv. | ||
|
||
Install `prefect-aws` | ||
|
||
```bash | ||
pip install prefect-aws | ||
``` | ||
|
||
### Registering blocks | ||
|
||
Register [blocks](https://docs.prefect.io/ui/blocks/) in this module to make them available for use. | ||
|
||
```bash | ||
prefect block register -m prefect_aws | ||
``` | ||
|
||
A list of available blocks in `prefect-aws` and their setup instructions can be found [here](https://PrefectHQ.github.io/prefect-aws/#blocks-catalog). | ||
|
||
### Saving credentials to a block | ||
|
||
You will need an AWS account and credentials to use `prefect-aws`. | ||
|
||
1. Refer to the [AWS Configuration documentation](https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-quickstart.html#cli-configure-quickstart-creds) on how to retrieve your access key ID and secret access key | ||
2. Copy the access key ID and secret access key | ||
3. Create an `AWSCredenitals` block in the Prefect UI or use a Python script like the one below and replace the placeholders with your credential information and desired block name: | ||
|
||
```python | ||
from prefect_aws import AwsCredentials | ||
AwsCredentials( | ||
aws_access_key_id="PLACEHOLDER", | ||
aws_secret_access_key="PLACEHOLDER", | ||
aws_session_token=None, # replace this with token if necessary | ||
region_name="us-east-2" | ||
).save("BLOCK-NAME-PLACEHOLDER") | ||
``` | ||
|
||
Congrats! You can now load the saved block to use your credentials in your Python code: | ||
|
||
```python | ||
from prefect_aws import AwsCredentials | ||
AwsCredentials.load("BLOCK-NAME-PLACEHOLDER") | ||
``` | ||
|
||
### Using Prefect with AWS S3 | ||
|
||
`prefect_aws` allows you to read and write objects with AWS S3 within your Prefect flows. | ||
|
||
The provided code snippet shows how you can use `prefect_aws` to upload a file to a AWS S3 bucket and download the same file under a different file name. | ||
|
||
Note, the following code assumes that the bucket already exists. | ||
|
||
```python | ||
from pathlib import Path | ||
from prefect import flow | ||
from prefect_aws import AwsCredentials, S3Bucket | ||
|
||
@flow | ||
def s3_flow(): | ||
# create a dummy file to upload | ||
file_path = Path("test-example.txt") | ||
file_path.write_text("Hello, Prefect!") | ||
|
||
aws_credentials = AwsCredentials.load("BLOCK-NAME-PLACEHOLDER") | ||
s3_bucket = S3Bucket( | ||
bucket_name="BUCKET-NAME-PLACEHOLDER", | ||
credentials=aws_credentials | ||
) | ||
|
||
s3_bucket_path = s3_bucket.upload_from_path(file_path) | ||
downloaded_file_path = s3_bucket.download_object_to_path( | ||
s3_bucket_path, "downloaded-test-example.txt" | ||
) | ||
return downloaded_file_path.read_text() | ||
|
||
s3_flow() | ||
``` | ||
|
||
### Using Prefect with AWS Secrets Manager | ||
|
||
`prefect_aws` allows you to read and write secrets with AWS Secrets Manager within your Prefect flows. | ||
|
||
The provided code snippet shows how you can use `prefect_aws` to write a secret to the Secret Manager, read the secret data, delete the secret, and finally return the secret data. | ||
|
||
```python | ||
from prefect import flow | ||
from prefect_aws import AwsCredentials, AwsSecret | ||
|
||
@flow | ||
def secrets_manager_flow(): | ||
aws_credentials = AwsCredentials.load("BLOCK-NAME-PLACEHOLDER") | ||
aws_secret = AwsSecret(secret_name="test-example", aws_credentials=aws_credentials) | ||
aws_secret.write_secret(secret_data=b"Hello, Prefect!") | ||
secret_data = aws_secret.read_secret() | ||
aws_secret.delete_secret() | ||
return secret_data | ||
|
||
secrets_manager_flow() | ||
``` | ||
|
||
### Using Prefect with AWS ECS | ||
|
||
`prefect_aws` allows you to use [AWS ECS](https://aws.amazon.com/ecs/) as infrastructure for your deployments. Using ECS for scheduled flow runs enables the dynamic provisioning of infrastructure for containers and unlocks greater scalability. This setup gives you all of the observation and orchestration benefits of Prefect, while also providing you the scalability of ECS. | ||
|
||
See the [ECS guide](/ecs_guide/) for a full walkthrough. | ||
|
||
## Resources | ||
|
||
Refer to the API documentation on the sidebar to explore all the capabilities of Prefect AWS! | ||
|
||
For more tips on how to use blocks and tasks in Prefect integration libraries, check out the [docs](https://docs.prefect.io/integrations/usage/)! | ||
|
||
For more information about how to use Prefect, please refer to the [Prefect documentation](https://docs.prefect.io/). | ||
|
||
If you encounter any bugs while using `prefect-aws`, feel free to open an issue in the [`prefect`](https://github.com/PrefectHQ/prefect) repository. | ||
|
||
If you have any questions or issues while using `prefect-aws`, you can find help in the [Prefect Slack community](https://prefect.io/slack). |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,6 @@ | ||
--- | ||
description: Module handling AWS Lambda functions | ||
notes: This documentation page is generated from source file docstrings. | ||
--- | ||
|
||
::: prefect_aws.lambda_function |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,6 @@ | ||
--- | ||
description: Tasks for interacting with AWS S3 | ||
notes: This documentation page is generated from source file docstrings. | ||
--- | ||
|
||
::: prefect_aws.s3 |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,6 @@ | ||
--- | ||
description: Tasks for interacting with AWS Secrets Manager | ||
notes: This documentation page is generated from source file docstrings. | ||
--- | ||
|
||
::: prefect_aws.secrets_manager |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,24 @@ | ||
# `prefect-aws` | ||
|
||
<p align="center"> | ||
<a href="https://pypi.python.org/pypi/prefect-aws/" alt="PyPI version"> | ||
<img alt="PyPI" src="https://img.shields.io/pypi/v/prefect-aws?color=26272B&labelColor=090422"></a> | ||
<a href="https://pepy.tech/badge/prefect-aws/" alt="Downloads"> | ||
<img src="https://img.shields.io/pypi/dm/prefect-aws?color=26272B&labelColor=090422" /></a> | ||
</p> | ||
|
||
## Welcome! | ||
|
||
`prefect-aws` makes it easy to leverage the capabilities of AWS in your flows, featuring support for ECS, S3, Secrets Manager, and Batch. | ||
|
||
### Installation | ||
|
||
To start using `prefect-aws`: | ||
|
||
```bash | ||
pip install prefect-aws | ||
``` | ||
|
||
### Contributing | ||
|
||
Thanks for thinking about chipping in! Check out this [step-by-step guide](https://prefecthq.github.io/prefect-aws/#installation) on how to get started. |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,29 @@ | ||
from . import _version | ||
from .credentials import AwsCredentials, MinIOCredentials | ||
from .client_parameters import AwsClientParameters | ||
from .lambda_function import LambdaFunction | ||
from .s3 import S3Bucket | ||
from .ecs import ECSTask | ||
from .secrets_manager import AwsSecret | ||
from .workers import ECSWorker | ||
|
||
from prefect._internal.compatibility.deprecated import ( | ||
register_renamed_module, | ||
) | ||
|
||
register_renamed_module( | ||
"prefect_aws.projects", "prefect_aws.deployments", start_date="Jun 2023" | ||
) | ||
|
||
__all__ = [ | ||
"AwsCredentials", | ||
"AwsClientParameters", | ||
"LambdaFunction", | ||
"MinIOCredentials", | ||
"S3Bucket", | ||
"ECSTask", | ||
"AwsSecret", | ||
"ECSWorker", | ||
] | ||
|
||
__version__ = _version.__version__ |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,73 @@ | ||
"""Tasks for interacting with AWS Batch""" | ||
|
||
from typing import Any, Dict, Optional | ||
|
||
from prefect import get_run_logger, task | ||
from prefect.utilities.asyncutils import run_sync_in_worker_thread | ||
from prefect_aws.credentials import AwsCredentials | ||
|
||
|
||
@task | ||
async def batch_submit( | ||
job_name: str, | ||
job_queue: str, | ||
job_definition: str, | ||
aws_credentials: AwsCredentials, | ||
**batch_kwargs: Optional[Dict[str, Any]], | ||
) -> str: | ||
""" | ||
Submit a job to the AWS Batch job service. | ||
Args: | ||
job_name: The AWS batch job name. | ||
job_queue: Name of the AWS batch job queue. | ||
job_definition: The AWS batch job definition. | ||
aws_credentials: Credentials to use for authentication with AWS. | ||
**batch_kwargs: Additional keyword arguments to pass to the boto3 | ||
`submit_job` function. See the documentation for | ||
[submit_job](https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/batch.html#Batch.Client.submit_job) | ||
for more details. | ||
Returns: | ||
The id corresponding to the job. | ||
Example: | ||
Submits a job to batch. | ||
```python | ||
from prefect import flow | ||
from prefect_aws import AwsCredentials | ||
from prefect_aws.batch import batch_submit | ||
@flow | ||
def example_batch_submit_flow(): | ||
aws_credentials = AwsCredentials( | ||
aws_access_key_id="acccess_key_id", | ||
aws_secret_access_key="secret_access_key" | ||
) | ||
job_id = batch_submit( | ||
"job_name", | ||
"job_queue", | ||
"job_definition", | ||
aws_credentials | ||
) | ||
return job_id | ||
example_batch_submit_flow() | ||
``` | ||
""" # noqa | ||
logger = get_run_logger() | ||
logger.info("Preparing to submit %s job to %s job queue", job_name, job_queue) | ||
|
||
batch_client = aws_credentials.get_boto3_session().client("batch") | ||
|
||
response = await run_sync_in_worker_thread( | ||
batch_client.submit_job, | ||
jobName=job_name, | ||
jobQueue=job_queue, | ||
jobDefinition=job_definition, | ||
**batch_kwargs, | ||
) | ||
return response["jobId"] |
Oops, something went wrong.