Migrate prefect-aws to core

docs/integrations/catalog/prefect-aws.yaml

@@ -1,6 +1,6 @@
 collectionName: prefect-aws
 author: Prefect
 authorUrl: https://prefect.io
-documentation: https://prefecthq.github.io/prefect-aws/
+documentation: prefect-aws
 iconUrl: /img/collections/aws.png
 tag: AWS

docs/integrations/prefect-aws/batch.md

@@ -0,0 +1,6 @@
+---
+description: Tasks for interacting with AWS Batch
+notes: This documentation page is generated from source file docstrings.
+---
+
+::: prefect_aws.batch

docs/integrations/prefect-aws/client_waiter.md

@@ -0,0 +1,6 @@
+---
+description: Task for waiting on a long-running AWS job
+notes: This documentation page is generated from source file docstrings.
+---
+
+::: prefect_aws.client_waiter

docs/integrations/prefect-aws/credentials.md

@@ -0,0 +1,6 @@
+---
+description: Module handling AWS credentials
+notes: This documentation page is generated from source file docstrings.
+---
+
+::: prefect_aws.credentials

docs/integrations/prefect-aws/deployments/steps.md

@@ -0,0 +1,6 @@
+---
+description: Prefect deployment steps for managing deployment code storage via AWS S3.
+notes: This documentation page is generated from source file docstrings.
+---
+
+::: prefect_aws.deployments.steps

docs/integrations/prefect-aws/ecs.md

@@ -0,0 +1,6 @@
+---
+description: Integrations with the Amazon Elastic Container Service.
+notes: This documentation page is generated from source file docstrings.
+---
+
+::: prefect_aws.ecs

docs/integrations/prefect-aws/ecs_worker.md

@@ -0,0 +1,6 @@
+---
+description: Worker integration with the Amazon Elastic Container Service.
+notes: This documentation page is generated from source file docstrings.
+---
+
+::: prefect_aws.workers.ecs_worker

docs/integrations/prefect-aws/glue_job.md

@@ -0,0 +1,6 @@
+---
+description: Tasks for interacting with AWS Glue Job
+notes: This documentation page is generated from source file docstrings.
+---
+
+::: prefect_aws.glue_job

docs/integrations/prefect-aws/index.md

@@ -0,0 +1,135 @@
+# `prefect-aws`
+
+<p align="center">
+    <a href="https://pypi.python.org/pypi/prefect-aws/" alt="PyPI version">
+        <img alt="PyPI" src="https://img.shields.io/pypi/v/prefect-aws?color=26272B&labelColor=090422"></a>
+    <a href="https://pepy.tech/badge/prefect-aws/" alt="Downloads">
+        <img src="https://img.shields.io/pypi/dm/prefect-aws?color=26272B&labelColor=090422" /></a>
+</p>
+
+## Welcome
+
+`prefect-aws` makes it easy to leverage the capabilities of AWS in your workflows.
+
+## Getting started
+
+### Installation
+
+Prefect requires Python 3.8 or newer.
+
+We recommend using a Python virtual environment manager such as pipenv, conda, or virtualenv.
+
+Install `prefect-aws`
+
+```bash
+pip install prefect-aws
+```
+
+### Registering blocks
+
+Register [blocks](https://docs.prefect.io/ui/blocks/) in this module to make them available for use.
+
+```bash
+prefect block register -m prefect_aws
+```
+
+A list of available blocks in `prefect-aws` and their setup instructions can be found [here](https://PrefectHQ.github.io/prefect-aws/#blocks-catalog).
+
+### Saving credentials to a block
+
+You will need an AWS account and credentials to use `prefect-aws`.
+
+1. Refer to the [AWS Configuration documentation](https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-quickstart.html#cli-configure-quickstart-creds) on how to retrieve your access key ID and secret access key
+2. Copy the access key ID and secret access key
+3. Create an `AWSCredenitals` block in the Prefect UI or use a Python script like the one below and replace the placeholders with your credential information and desired block name:
+
+```python
+from prefect_aws import AwsCredentials
+AwsCredentials(
+    aws_access_key_id="PLACEHOLDER",
+    aws_secret_access_key="PLACEHOLDER",
+    aws_session_token=None,  # replace this with token if necessary
+    region_name="us-east-2"
+).save("BLOCK-NAME-PLACEHOLDER")
+```
+
+Congrats! You can now load the saved block to use your credentials in your Python code:
+
+```python
+from prefect_aws import AwsCredentials
+AwsCredentials.load("BLOCK-NAME-PLACEHOLDER")
+```
+
+### Using Prefect with AWS S3
+
+`prefect_aws` allows you to read and write objects with AWS S3 within your Prefect flows.
+
+The provided code snippet shows how you can use `prefect_aws` to upload a file to a AWS S3 bucket and download the same file under a different file name.
+
+Note, the following code assumes that the bucket already exists.
+
+```python
+from pathlib import Path
+from prefect import flow
+from prefect_aws import AwsCredentials, S3Bucket
+
+@flow
+def s3_flow():
+    # create a dummy file to upload
+    file_path = Path("test-example.txt")
+    file_path.write_text("Hello, Prefect!")
+
+    aws_credentials = AwsCredentials.load("BLOCK-NAME-PLACEHOLDER")
+    s3_bucket = S3Bucket(
+        bucket_name="BUCKET-NAME-PLACEHOLDER",
+        credentials=aws_credentials
+    )
+
+    s3_bucket_path = s3_bucket.upload_from_path(file_path)
+    downloaded_file_path = s3_bucket.download_object_to_path(
+        s3_bucket_path, "downloaded-test-example.txt"
+    )
+    return downloaded_file_path.read_text()
+
+s3_flow()
+```
+
+### Using Prefect with AWS Secrets Manager
+
+`prefect_aws` allows you to read and write secrets with AWS Secrets Manager within your Prefect flows.
+
+The provided code snippet shows how you can use `prefect_aws` to write a secret to the Secret Manager, read the secret data, delete the secret, and finally return the secret data.
+
+```python
+from prefect import flow
+from prefect_aws import AwsCredentials, AwsSecret
+
+@flow
+def secrets_manager_flow():
+    aws_credentials = AwsCredentials.load("BLOCK-NAME-PLACEHOLDER")
+    aws_secret = AwsSecret(secret_name="test-example", aws_credentials=aws_credentials)
+    aws_secret.write_secret(secret_data=b"Hello, Prefect!")
+    secret_data = aws_secret.read_secret()
+    aws_secret.delete_secret()
+    return secret_data
+
+secrets_manager_flow()
+```
+
+### Using Prefect with AWS ECS
+
+`prefect_aws` allows you to use [AWS ECS](https://aws.amazon.com/ecs/) as infrastructure for your deployments. Using ECS for scheduled flow runs enables the dynamic provisioning of infrastructure for containers and unlocks greater scalability. This setup gives you all of the observation and orchestration benefits of Prefect, while also providing you the scalability of ECS.
+
+See the [ECS guide](/ecs_guide/) for a full walkthrough.
+
+## Resources
+
+Refer to the API documentation on the sidebar to explore all the capabilities of Prefect AWS!
+
+For more tips on how to use blocks and tasks in Prefect integration libraries, check out the [docs](https://docs.prefect.io/integrations/usage/)!
+
+For more information about how to use Prefect, please refer to the [Prefect documentation](https://docs.prefect.io/).
+
+If you encounter any bugs while using `prefect-aws`, feel free to open an issue in the [`prefect`](https://github.com/PrefectHQ/prefect) repository.
+
+If you have any questions or issues while using `prefect-aws`, you can find help in the [Prefect Slack community](https://prefect.io/slack).

docs/integrations/prefect-aws/lambda_function.md

@@ -0,0 +1,6 @@
+---
+description: Module handling AWS Lambda functions
+notes: This documentation page is generated from source file docstrings.
+---
+
+::: prefect_aws.lambda_function

docs/integrations/prefect-aws/s3.md

@@ -0,0 +1,6 @@
+---
+description: Tasks for interacting with AWS S3
+notes: This documentation page is generated from source file docstrings.
+---
+
+::: prefect_aws.s3

docs/integrations/prefect-aws/secrets_manager.md

@@ -0,0 +1,6 @@
+---
+description: Tasks for interacting with AWS Secrets Manager
+notes: This documentation page is generated from source file docstrings.
+---
+
+::: prefect_aws.secrets_manager

mkdocs.yml

@@ -238,6 +238,21 @@ nav:
           - Using Integrations: integrations/usage.md
           - Contributing Integrations: integrations/contribute.md
           - Libraries:
+                - AWS:
+                      - integrations/prefect-aws/index.md
+                      - Guides:
+                        - Setting up an ECS Worker: integrations/prefect-aws/ecs_guide.md
+                      - Batch: integrations/prefect-aws/batch.md
+                      - Client Waiter: integrations/prefect-aws/client_waiter.md
+                      - Credentials: integrations/prefect-aws/credentials.md
+                      - ECS Worker: integrations/prefect-aws/ecs_worker.md
+                      - ECS (deprecated): integrations/prefect-aws/ecs.md
+                      - Glue Job: integrations/prefect-aws/glue_job.md
+                      - Lambda: integrations/prefect-aws/lambda_function.md
+                      - Deployments:
+                        - Steps: integrations/prefect-aws/deployments/steps.md
+                      - S3: integrations/prefect-aws/s3.md
+                      - Secrets Manager: integrations/prefect-aws/secrets_manager.md
                 - Azure:
                       - integrations/prefect-azure/index.md
                       - ACI Worker Guide: integrations/prefect-azure/aci_worker.md

scripts/serve_docs

@@ -30,12 +30,9 @@ cd "$(dirname "$0")/.."
 uv venv "$temp_dir/.venv"
 source "$temp_dir/.venv/bin/activate"
 
-# Install all integration libraries from src/integrations
+# Install all integration libraries from src/integrations and core dev dependencies as editable
 integration_packages=$(find src/integrations -name "pyproject.toml" -exec dirname {} \;)
-uv pip install $integration_packages
-
-# Install the prefect package in editable mode
-uv pip install -e ".[dev]"
+uv pip install $integration_packages -e ".[dev]"
 
 # Build and serve the docs
 mkdocs serve -a localhost:8000

src/integrations/prefect-aws/README.md

@@ -0,0 +1,24 @@
+# `prefect-aws`
+
+<p align="center">
+    <a href="https://pypi.python.org/pypi/prefect-aws/" alt="PyPI version">
+        <img alt="PyPI" src="https://img.shields.io/pypi/v/prefect-aws?color=26272B&labelColor=090422"></a>
+    <a href="https://pepy.tech/badge/prefect-aws/" alt="Downloads">
+        <img src="https://img.shields.io/pypi/dm/prefect-aws?color=26272B&labelColor=090422" /></a>
+</p>
+
+## Welcome!
+
+`prefect-aws` makes it easy to leverage the capabilities of AWS in your flows, featuring support for ECS, S3, Secrets Manager, and Batch.
+
+### Installation
+
+To start using `prefect-aws`:
+
+```bash
+pip install prefect-aws
+```
+
+### Contributing
+
+Thanks for thinking about chipping in! Check out this [step-by-step guide](https://prefecthq.github.io/prefect-aws/#installation) on how to get started.

src/integrations/prefect-aws/prefect_aws/__init__.py

@@ -0,0 +1,29 @@
+from . import _version
+from .credentials import AwsCredentials, MinIOCredentials
+from .client_parameters import AwsClientParameters
+from .lambda_function import LambdaFunction
+from .s3 import S3Bucket
+from .ecs import ECSTask
+from .secrets_manager import AwsSecret
+from .workers import ECSWorker
+
+from prefect._internal.compatibility.deprecated import (
+    register_renamed_module,
+)
+
+register_renamed_module(
+    "prefect_aws.projects", "prefect_aws.deployments", start_date="Jun 2023"
+)
+
+__all__ = [
+    "AwsCredentials",
+    "AwsClientParameters",
+    "LambdaFunction",
+    "MinIOCredentials",
+    "S3Bucket",
+    "ECSTask",
+    "AwsSecret",
+    "ECSWorker",
+]
+
+__version__ = _version.__version__

src/integrations/prefect-aws/prefect_aws/batch.py

@@ -0,0 +1,73 @@
+"""Tasks for interacting with AWS Batch"""
+
+from typing import Any, Dict, Optional
+
+from prefect import get_run_logger, task
+from prefect.utilities.asyncutils import run_sync_in_worker_thread
+from prefect_aws.credentials import AwsCredentials
+
+
+@task
+async def batch_submit(
+    job_name: str,
+    job_queue: str,
+    job_definition: str,
+    aws_credentials: AwsCredentials,
+    **batch_kwargs: Optional[Dict[str, Any]],
+) -> str:
+    """
+    Submit a job to the AWS Batch job service.
+
+    Args:
+        job_name: The AWS batch job name.
+        job_queue: Name of the AWS batch job queue.
+        job_definition: The AWS batch job definition.
+        aws_credentials: Credentials to use for authentication with AWS.
+        **batch_kwargs: Additional keyword arguments to pass to the boto3
+            `submit_job` function. See the documentation for
+            [submit_job](https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/batch.html#Batch.Client.submit_job)
+            for more details.
+
+    Returns:
+        The id corresponding to the job.
+
+    Example:
+        Submits a job to batch.
+
+        ```python
+        from prefect import flow
+        from prefect_aws import AwsCredentials
+        from prefect_aws.batch import batch_submit
+
+
+        @flow
+        def example_batch_submit_flow():
+            aws_credentials = AwsCredentials(
+                aws_access_key_id="acccess_key_id",
+                aws_secret_access_key="secret_access_key"
+            )
+            job_id = batch_submit(
+                "job_name",
+                "job_queue",
+                "job_definition",
+                aws_credentials
+            )
+            return job_id
+
+        example_batch_submit_flow()
+        ```
+
+    """  # noqa
+    logger = get_run_logger()
+    logger.info("Preparing to submit %s job to %s job queue", job_name, job_queue)
+
+    batch_client = aws_credentials.get_boto3_session().client("batch")
+
+    response = await run_sync_in_worker_thread(
+        batch_client.submit_job,
+        jobName=job_name,
+        jobQueue=job_queue,
+        jobDefinition=job_definition,
+        **batch_kwargs,
+    )
+    return response["jobId"]