From 06a8dbfaaa7bbd4f422f4362e886598a8df9c512 Mon Sep 17 00:00:00 2001 From: Jeff Hale Date: Fri, 17 Nov 2023 18:45:02 -0500 Subject: [PATCH] Update ecs_guide.md formatting Standardize to match Prefect style guide and provide ability to link to subsections of the guide. --- docs/ecs_guide.md | 338 ++++++++++++++++++++++------------------------ 1 file changed, 162 insertions(+), 176 deletions(-) diff --git a/docs/ecs_guide.md b/docs/ecs_guide.md index d188a89a..8740c716 100644 --- a/docs/ecs_guide.md +++ b/docs/ecs_guide.md @@ -9,12 +9,12 @@ ECS (Elastic Container Service) tasks are a good option for executing Prefect 2 3. **AWS Integration**: Easily connect with other AWS services, such as AWS IAM and CloudWatch. 4. **Containerization**: ECS supports Docker containers and offers managed execution. Containerization encourages reproducible deployments. -## ECS Flow Run Execution +## ECS flow run execution Prefect enables remote flow execution via [workers](https://docs.prefect.io/concepts/work-pools/#worker-overview) and [work pools](https://docs.prefect.io/concepts/work-pools/#work-pool-overview). To learn more about these concepts please see our [deployment tutorial](https://docs.prefect.io/tutorial/deployments/). -For details on how workers and work pools are implemented for ECS, see the diagram below: -#### Architecture Diagram +For details on how workers and work pools are implemented for ECS, see the diagram below. + ```mermaid graph TB @@ -48,7 +48,8 @@ graph TB prefect_worker -->|polls| workqueue prefect_workpool -->|configures| fr_task_definition ``` -### ECS in Prefect Terms + +## ECS and Prefect !!! tip "ECS tasks != Prefect tasks" An ECS task is **not** the same thing as a [Prefect task](https://docs.prefect.io/latest/concepts/tasks/#tasks-overview). @@ -72,20 +73,17 @@ You can use either EC2 or Fargate as the capacity provider. Fargate simplifies i
-# AWS CLI Guide - !!! tip If you prefer infrastructure as code check out this [Terraform module](https://github.com/PrefectHQ/prefect-recipes/tree/main/devops/infrastructure-as-code/aws/tf-prefect2-ecs-worker) to provision an ECS cluster with a worker. -### Prerequisites -Before you begin, make sure you have: +## Prerequisites - An AWS account with permissions to create ECS services and IAM roles. - The AWS CLI installed on your local machine. You can [download it from the AWS website](https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html). - An [ECS Cluster](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/clusters.html) to host both the worker and the flow runs it submits. Follow [this guide](https://docs.aws.amazon.com/AmazonECS/latest/userguide/create_cluster.html) to create an ECS cluster or simply use the default cluster. - A [VPC](https://docs.aws.amazon.com/vpc/latest/userguide/what-is-amazon-vpc.html) configured for your ECS tasks. A VPC is a good idea if using EC2 and required if using Fargate. -### Step 1: Set Up an ECS work pool +## Step 1: Set Up an ECS work pool Before setting up the worker, create a simple [work pool](https://docs.prefect.io/latest/concepts/work-pools/#work-pool-configuration) of type ECS for the worker to pull work from. @@ -109,206 +107,194 @@ Configuring custom fields is easiest from the UI. Next, set up a Prefect ECS worker that will discover and pull work from this work pool. -### Step 2: Start a Prefect worker in your ECS cluster. - +## Step 2: Start a Prefect worker in your ECS cluster To create an [IAM role](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_create_for-custom.html#roles-creatingrole-custom-trust-policy-console) for the ECS task using the AWS CLI, follow these steps: -1. **Create a trust policy** +### 1. Create a trust policy - The trust policy will specify that ECS can assume the role. +The trust policy will specify that ECS can assume the role. - Save this policy to a file, such as `ecs-trust-policy.json`: +Save this policy to a file, such as `ecs-trust-policy.json`: - ```json +```json - { - "Version": "2012-10-17", - "Statement": [ - { - "Effect": "Allow", - "Principal": { - "Service": "ecs-tasks.amazonaws.com" - }, - "Action": "sts:AssumeRole" - } - ] - } - ``` - -2. **Create the IAM role** - - Use the `aws iam create-role` command to create the role: - - ```bash - - aws iam create-role \ - --role-name ecsTaskExecutionRole \ - --assume-role-policy-document file://ecs-trust-policy.json - ``` - -3. **Attach the policy to the role** - - Amazon has a managed policy named `AmazonECSTaskExecutionRolePolicy` that grants the permissions necessary for ECS tasks. Attach this policy to your role: - - ```bash - - aws iam attach-role-policy \ - --role-name ecsTaskExecutionRole \ - --policy-arn arn:aws:iam::aws:policy/service-role/AmazonECSTaskExecutionRolePolicy - ``` - - Remember to replace the `--role-name` and `--policy-arn` with the actual role name and policy Amazon Resource Name (ARN) you want to use. - - Now, you have a role named `ecsTaskExecutionRole` that you can assign to your ECS tasks. This role has the necessary permissions to pull container images and publish logs to CloudWatch. - -4. **Launch an ECS Service to host the worker** - - Next, create an ECS task definition that specifies the Docker image for the Prefect worker, the resources it requires, and the command it should run. In this example, the command to start the worker is `prefect worker start --pool my-ecs-pool`. - - **Create a JSON file with the following contents:** - - ```json - { - "family": "prefect-worker-task", - "networkMode": "awsvpc", - "requiresCompatibilities": [ - "FARGATE" - ], - "cpu": "512", - "memory": "1024", - "executionRoleArn": "", - "taskRoleArn": "", - "containerDefinitions": [ - { - "name": "prefect-worker", - "image": "prefecthq/prefect:2-latest", - "cpu": 512, - "memory": 1024, - "essential": true, - "command": [ - "/bin/sh", - "-c", - "pip install prefect-aws && prefect worker start --pool my-ecs-pool --type ecs" - ], - "environment": [ - { - "name": "PREFECT_API_URL", - "value": "https://api.prefect.cloud/api/accounts//workspaces/" - }, - { - "name": "PREFECT_API_KEY", - "value": "" - } - ] - } - ] - } - ``` - - - Use `prefect config view` to view the `PREFECT_API_URL` for your current Prefect profile. Use this to replace both `` and ``. - - - For the `PREFECT_API_KEY`, individuals on the organization tier can create a [service account](https://docs.prefect.io/latest/cloud/users/service-accounts/) for the worker. If on a personal tier, you can pass a user’s API key. - - - Replace both instances of `` with the ARN of the IAM role you created in Step 2. - - - Notice that the CPU and Memory allocations are relatively small. The worker's main responsibility is to submit work through API calls to AWS, _not_ to execute your Prefect flow code. - - !!! tip - To avoid hardcoding your API key into the task definition JSON see [how to add environment variables to the container definition](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/secrets-envvar-secrets-manager.html#secrets-envvar-secrets-manager-update-container-definition). The API key must be stored as plain text, not the key-value pair dictionary that it is formatted in by default. - -6. **Register the task definition:** - - Before creating a service, you first need to register a task definition. You can do that using the `register-task-definition` command in the AWS CLI. Here is an example: +{ + "Version": "2012-10-17", + "Statement": [ + { + "Effect": "Allow", + "Principal": { + "Service": "ecs-tasks.amazonaws.com" + }, + "Action": "sts:AssumeRole" + } + ] +} +``` - ```bash +### 2. Create the IAM role - aws ecs register-task-definition --cli-input-json file://task-definition.json - ``` +Use the `aws iam create-role` command to create the role: + +```bash +aws iam create-role \ +--role-name ecsTaskExecutionRole \ +--assume-role-policy-document file://ecs-trust-policy.json +``` + +### 3. Attach the policy to the role + +Amazon has a managed policy named `AmazonECSTaskExecutionRolePolicy` that grants the permissions necessary for ECS tasks. Attach this policy to your role: + +```bash +aws iam attach-role-policy \ +--role-name ecsTaskExecutionRole \ +--policy-arn arn:aws:iam::aws:policy/service-role/AmazonECSTaskExecutionRolePolicy +``` +Remember to replace the `--role-name` and `--policy-arn` with the actual role name and policy Amazon Resource Name (ARN) you want to use. + +Now, you have a role named `ecsTaskExecutionRole` that you can assign to your ECS tasks. This role has the necessary permissions to pull container images and publish logs to CloudWatch. + +### 4. Launch an ECS Service to host the worker + +Next, create an ECS task definition that specifies the Docker image for the Prefect worker, the resources it requires, and the command it should run. In this example, the command to start the worker is `prefect worker start --pool my-ecs-pool`. + +Create a JSON file with the following contents: + +```json +{ + "family": "prefect-worker-task", + "networkMode": "awsvpc", + "requiresCompatibilities": [ + "FARGATE" + ], + "cpu": "512", + "memory": "1024", + "executionRoleArn": "", + "taskRoleArn": "", + "containerDefinitions": [ + { + "name": "prefect-worker", + "image": "prefecthq/prefect:2-latest", + "cpu": 512, + "memory": 1024, + "essential": true, + "command": [ + "/bin/sh", + "-c", + "pip install prefect-aws && prefect worker start --pool my-ecs-pool --type ecs" + ], + "environment": [ + { + "name": "PREFECT_API_URL", + "value": "https://api.prefect.cloud/api/accounts//workspaces/" + }, + { + "name": "PREFECT_API_KEY", + "value": "" + } + ] + } + ] +} +``` +- Use `prefect config view` to view the `PREFECT_API_URL` for your current Prefect profile. Use this to replace both `` and ``. +- For the `PREFECT_API_KEY`, individuals on the organization tier can create a [service account](https://docs.prefect.io/latest/cloud/users/service-accounts/) for the worker. If on a personal tier, you can pass a user’s API key. +- Replace both instances of `` with the ARN of the IAM role you created in Step 2. +- Notice that the CPU and Memory allocations are relatively small. The worker's main responsibility is to submit work through API calls to AWS, _not_ to execute your Prefect flow code. - Replace `task-definition.json` with the name of your JSON file. +!!! tip + To avoid hardcoding your API key into the task definition JSON see [how to add environment variables to the container definition](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/secrets-envvar-secrets-manager.html#secrets-envvar-secrets-manager-update-container-definition). The API key must be stored as plain text, not the key-value pair dictionary that it is formatted in by default. -7. **Create an ECS service to host your worker:** +### 5. Register the task definition + +Before creating a service, you first need to register a task definition. You can do that using the `register-task-definition` command in the AWS CLI. Here is an example: - Finally, create a service that will manage your Prefect worker: +```bash +aws ecs register-task-definition --cli-input-json file://task-definition.json +``` - Open a terminal window and run the following command to create an ECS Fargate service: +Replace `task-definition.json` with the name of your JSON file. - ```bash - aws ecs create-service \ - --service-name prefect-worker-service \ - --cluster \ - --task-definition \ - --launch-type FARGATE \ - --desired-count 1 \ - --network-configuration "awsvpcConfiguration={subnets=[],securityGroups=[]}" - ``` +### 6. Create an ECS service to host your worker - - Replace `` with the name of your ECS cluster. - - Replace `` with the path to the JSON file you created in Step 2, `` with a comma-separated list of your VPC subnet IDs. Ensure that these subnets are aligned with the vpc specified on the work pool in step 1. - - Replace `` with a comma-separated list of your VPC security group IDs. - - Replace `` with the ARN of the task definition you just registered. +Finally, create a service that will manage your Prefect worker: - !!! tip "Sanity check" - The work pool page in the Prefect UI allows you to check the health of your workers - make sure your new worker is live! +Open a terminal window and run the following command to create an ECS Fargate service: -### Step 4: Pick up a flow run with your new worker! +```bash +aws ecs create-service \ + --service-name prefect-worker-service \ + --cluster \ + --task-definition \ + --launch-type FARGATE \ + --desired-count 1 \ + --network-configuration "awsvpcConfiguration={subnets=[],securityGroups=[]}" +``` -1. Write a simple test flow in a repo of your choice: +- Replace `` with the name of your ECS cluster. +- Replace `` with the path to the JSON file you created in Step 2, `` with a comma-separated list of your VPC subnet IDs. Ensure that these subnets are aligned with the vpc specified on the work pool in step 1. +- Replace `` with a comma-separated list of your VPC security group IDs. +- Replace `` with the ARN of the task definition you just registered. - `my_flow.py` +!!! tip "Sanity check" + The work pool page in the Prefect UI allows you to check the health of your workers - make sure your new worker is live! - ```python - from prefect import flow, get_run_logger +## Step 3: Pick up a flow run with your new worker! - @flow - def my_flow(): - logger = get_run_logger() - logger.info("Hello from ECS!!") +### 1. Write a simple test flow in a repo of your choice: - if __name__ == "__main__": - my_flow() - ``` +```python title="my_flow.py" +from prefect import flow, get_run_logger -2. [Deploy](https://docs.prefect.io/2.11.0/tutorial/deployments/#create-a-deployment) the flow to the server, specifying the ECS work pool when prompted. +@flow +def my_flow(): + logger = get_run_logger() + logger.info("Hello from ECS!!") - ```bash - prefect deploy my_flow.py:my_flow - ``` +if __name__ == "__main__": + my_flow() +``` -3. Find the deployment in the UI and click the **Quick Run** button! +### 2. [Deploy](https://docs.prefect.io/2.11.0/tutorial/deployments/#create-a-deployment) the flow to the server, specifying the ECS work pool when prompted. +```bash +prefect deploy my_flow.py:my_flow +``` -## Optional Next Steps +### 3. Find the deployment in the UI and click the **Quick Run** button! -1. Now that you are confident your ECS worker is healthy, you can experiment with different work pool configurations. +## Next steps - - Do your flow runs require higher `CPU`? - - Would an EC2 `Launch Type` speed up your flow run execution? +Now that you are confident your ECS worker is healthy, you can experiment with different work pool configurations. - These infrastructure configuration values can be set on your ECS work pool or they can be overridden on the deployment level through [job_variables](https://docs.prefect.io/concepts/infrastructure/#kubernetesjob-overrides-and-customizations) if desired. +- Do your flow runs require higher `CPU`? +- Would an EC2 `Launch Type` speed up your flow run execution? +These infrastructure configuration values can be set on your ECS work pool or they can be overridden on the deployment level through [job_variables](https://docs.prefect.io/concepts/infrastructure/#kubernetesjob-overrides-and-customizations) if desired. -2. Consider adding a [build action](https://docs.prefect.io/concepts/deployments-ux/#the-build-action) to your Prefect Project [`prefect.yaml`](https://docs.prefect.io/concepts/deployments-ux/#the-prefect-yaml-file) if you want to automatically build a Docker image and push it to an image registry `prefect deploy` is run. +Consider adding a [build action](https://docs.prefect.io/concepts/deployments-ux/#the-build-action) to your [`prefect.yaml`](https://docs.prefect.io/concepts/deployments-ux/#the-prefect-yaml-file) file if you want to automatically build a Docker image and push it to an image registry. Note that a Docker image is built and pushed by default if a deployemnt is creaded with the `flow.deploy` method in Python. Here is an example build action for ECR: - ```yaml - build: - - prefect.deployments.steps.run_shell_script: - id: get-commit-hash - script: git rev-parse --short HEAD - stream_output: false - - prefect.deployments.steps.run_shell_script: - id: ecr-auth-step - script: aws ecr get-login-password --region | docker login --username - AWS --password-stdin <>.dkr.ecr..amazonaws.com - stream_output: false - - prefect_docker.deployments.steps.build_docker_image: - requires: prefect-docker>=0.3.0 - image_name: .dkr.ecr.us-east-2.amazonaws.com/ - tag: '{{ get-commit-hash.stdout }}' - dockerfile: auto - push: true - ``` + +```yaml +build: +- prefect.deployments.steps.run_shell_script: + id: get-commit-hash + script: git rev-parse --short HEAD + stream_output: false +- prefect.deployments.steps.run_shell_script: + id: ecr-auth-step + script: aws ecr get-login-password --region | docker login --username + AWS --password-stdin <>.dkr.ecr..amazonaws.com + stream_output: false +- prefect_docker.deployments.steps.build_docker_image: + requires: prefect-docker>=0.3.0 + image_name: .dkr.ecr.us-east-2.amazonaws.com/ + tag: '{{ get-commit-hash.stdout }}' + dockerfile: auto + push: true +```