Skip to content

Latest commit

 

History

History
199 lines (162 loc) · 11.8 KB

deployment.md

File metadata and controls

199 lines (162 loc) · 11.8 KB
Raw

Deploying the Metaflowbot

Deploying the Metaflowbot involves -

  1. Setting up the Metaflowbot on Slack, and
  2. Running the Metaflowbot server

Setting up the Metaflowbot on Slack

  1. Create an App on Slack UI using the provided manifest. The default name of the Metaflowbot is @flowey. To customize the name of the Metaflowbot, change display_information.name and bot_user.display_name in the manifest.

  2. Install the App

  3. Generate an App token (SLACK_APP_TOKEN): This token allows the Metaflowbot to make a socket connection to Slack and will be used later to configure the bot.

  4. Generate Bot token (SLACK_BOT_TOKEN) : This token allows the Metaflowbot to make web API calls and will be used later to configure the bot.

  5. Here is a walkthrough ezgif-6-e492971fc733

Running the Metaflowbot server

locally with pip

The Metaflowbot server is available as a pip package from PyPI and can be directly invoked.

  1. Install metaflowbot Python package from PyPI

    pip install metaflowbot
pip install metaflowbot-actions-jokes # Optional dependency

  2. Launch the Metaflowbot server by providing --admin argument with the email address of your slack account; Metaflowbot will open a message thread with you to maintain it's state (as a poor man's database). Replace SLACK_APP_TOKEN & SLACK_BOT_TOKEN with the values obtained while setting up the Metaflowbot on Slack.

    SLACK_APP_TOKEN=xapp-foo SLACK_BOT_TOKEN=xoxb-bar python -m metaflowbot server --admin me@server.com

locally via a docker image

The Metaflowbot server is also available as a docker image from Docker Hub. There are multiple ways to configure the image; just ensure that ADMIN_USER_ADDRESS environment variable points to your email address in the Slack workspace -

  • through environment variables
docker run -i -t --rm \
    -e SLACK_BOT_TOKEN=$(echo $SLACK_BOT_TOKEN) \
    -e ADMIN_USER_ADDRESS=admin@server.com \
    -e SLACK_APP_TOKEN=$(echo $SLACK_APP_TOKEN) \
    -e AWS_SECRET_ACCESS_KEY=$(echo $AWS_SECRET_ACCESS_KEY) \
    -e AWS_ACCESS_KEY_ID=$(echo $AWS_ACCESS_KEY_ID) \
    -e USERNAME=metaflowbot \
    -e METAFLOW_SERVICE_AUTH_KEY=$(echo $METAFLOW_SERVICE_AUTH_KEY) \
    -e METAFLOW_SERVICE_URL=$(echo $METAFLOW_SERVICE_URL) \
    -e METAFLOW_DATASTORE_SYSROOT_S3=$(echo $METAFLOW_DATASTORE_SYSROOT_S3) \
    -e METAFLOW_DEFAULT_DATASTORE=s3 \
    -e METAFLOW_DEFAULT_METADATA=service \
    outerbounds/metaflowbot
  • through ~/.metaflowconfig.
docker run -it \
    -v ~/.metaflowconfig:/metaflowconfig --rm \
    -e SLACK_BOT_TOKEN=$(echo $SLACK_BOT_TOKEN) \
    -e ADMIN_USER_ADDRESS=admin@server.com \
    -e SLACK_APP_TOKEN=$(echo $SLACK_APP_TOKEN) \
    -e AWS_SECRET_ACCESS_KEY=$(echo $AWS_SECRET_ACCESS_KEY) \
    -e AWS_ACCESS_KEY_ID=$(echo $AWS_ACCESS_KEY_ID) \
    -e USERNAME=metaflowbot \
    -e METAFLOW_HOME=/.metaflowconfig \
    outerbounds/metaflowbot

on AWS with AWS CloudFormation

Metaflow bot ships with an AWS CloudFormation template that automates the deployment of all the necessary AWS resources. The template is provided in the deployment folder.

The major components of the template are:

  1. AWS Identity and Access Management - Set policies for accessing cloud resources and secrets needed for deployment.
  2. AWS VPC Networking - A VPC with public subnet and internet gateway to deploy Metaflowbot.
  3. AWS VPC Security Groups - Outbound traffic access for Metaflowbot's container.
  4. AWS ECS - Deploying the Metaflowbot's container as a Fargate task.
  5. AWS SecretsManager - Access to secrets holding authentication information about Slack and Metadata service.

Deploying the template requires a few auth tokens (for Slack and Metaflow Service); these need to be created in AWS Secrets Manager which are referenced in the CloudFormation template.

  1. In your AWS Console for AWS Secrets Manager, create a secret with the auth tokens for Slack (SLACK_APP_TOKEN, SLACK_BOT_TOKEN, ) and Metaflow Service (METAFLOW_SERVICE_AUTH_KEY). Copy the ARN of the secret

  2. Paste the ARN of the secret along with other metadata + s3 related deployment details.

on AWS manually

If you cannot use the AWS CloudFormation template, follow these steps for a manual deployment of Metaflowbot on AWS.

Please note that Metaflow bot can re-use existing AWS resources - for example, your existing ECS cluster for container deployment. The instructions listed here will create these resources from scratch. If you have a strong background in administering AWS resources, you will notice that many of the security policies are fairly permissive and are intended to serve as a starting point for more complex deployments. Please reach out to us if you would like to discuss more involved deployments.

VPC

  1. Open the Amazon VPC console and in the left navigation pane, choose VPC Dashboard.
  2. Choose Launch VPC Wizard,
  3. Choose VPC with a Single Public Subnet and press Select.
  4. For VPC name, give your VPC a unique name.
  5. Choose Create VPC.
  6. When the wizard is finished, choose OK.

ECS Execution IAM Role (Optional)

  1. Open the IAM console and in the navigation pane, choose Roles, Create role.
  2. For Select type of trusted entity section, choose AWS service.
  3. For Choose the service that will use this role, choose Elastic Container Service.
  4. For Select your use case, choose Elastic Container Service Task and choose Next: Permissions.
  5. Choose AmazonECSTaskExecutionRolePolicy.
  6. Choose Next:tags.
  7. For Add tags (optional), enter any metadata tags you want to associate with the IAM role, and then choose Next: Review.
  8. For Role name, enter a name for your role and then choose Create role to finish. Note the ARN of the IAM role you just created.

ECS Task IAM Role

  1. Open the IAM console and in the navigation pane, choose Roles, Create role.
  2. For Select type of trusted entity section, choose AWS service.
  3. For Choose the service that will use this role, choose Elastic Container Service.
  4. For Select your use case, choose Elastic Container Service Task and choose Next: Permissions.
  5. Next, we will create a policy for Amazon S3 and attach it to this role:
    1. Amazon S3 for data storage
      1. Choose Create Policy to open a new window.
      2. Use the visual service editor to create the policy
        1. For Service, choose S3.
        2. For Actions, add GetObject and ListBucket as allowed actions
        3. For resources, the bucket name should be the same as metaflow datastore S3 bucket. For object choose any for object name. Choose Save changes.
        4. Choose Review policy. On the Review policy page, for Name type your own unique name and choose Create policy to finish.
  6. Click the refresh button in the original pane (in Step 4.) and choose the policy that you just created (in Step 5.). Choose Next:tags.
  7. For Add tags (optional), enter any metadata tags you want to associate with the IAM role, and then choose Next: Review.
  8. For Role name, enter a name for your role and then choose Create role to finish.

ECS Cluster + Fargate Task

  1. Open the ECS console and from the navigation bar, select the region to use.
  2. Choose Create Cluster under Clusters.
  3. Choose Networking only, Next step.
  4. Pick a name for Cluster name. Don't enable Create VPC. We will use the VPC we have created previously. You can choose to check Enable Container Insights. Choose Create.
  5. Choose View Cluster and choose Task Definitions on the left side pane.
  6. Choose Create new Task Definition, Fargate and Next step.
    1. Under Configure task and container definitions,
      1. Choose a Task Definition Name.
      2. Choose the Task Role as the one you just created above.
    2. Under Task execution IAM role, set the Task execution role to ecsTaskExecutionRole or set it to the IAM role created for ECS execution. Leave it empty otherwise.
    3. Under Task size,
      1. Choose 8 GB for Task memory (GB)
      2. Choose 4 vCPU for Task CPU (vCPU).
    4. Under Container Definitions, choose Add container
      1. Set metaflowbot as the Container name.
      2. Set outerbounds/metaflowbot as the Image.
      3. Leave other options as is.
      4. Under Advanced container configuration, in Environment variables add the following values
        1. Set Key as ADMIN_USER_ADDRESS and the Value as the email address of the user in the slack workspace with whom the bot will open a message thread to store state related information.
        2. Set Key as METAFLOW_SERVICE_URL and the Value as the URL to the metadata service.
        3. Set Key as METAFLOW_DATASTORE_SYSROOT_S3 and the Value as S3 bucket URL for metaflow datastore.
        4. Set Key as METAFLOW_DEFAULT_DATASTORE and Value as s3.
        5. Set Key as METAFLOW_DEFAULT_METADATA and Value as service.
        6. Set Key as USERNAME and Value as slackbot.
        7. Set Key as SLACK_APP_TOKEN and Value as the SLACK_APP_TOKEN retrieved from [Slack].
        8. Set Key as SLACK_BOT_TOKEN and Value as the SLACK_BOT_TOKEN retrieved from [Slack].
        9. If your metadata service has an authentication key to it then Set Key as METAFLOW_SERVICE_AUTH_KEY and value as the authentication token of the metadata service.
      5. Choose Add.
    5. Choose Create.
  7. Choose Clusters in the left side pane and select the cluster you created in Step 4.
  8. Choose Create under Services,
    1. Choose Fargate as Lauch type.
    2. Choose the task definition that you created in Step 6. for Task Definition. Pick the latest for Revision.
    3. For Platform version choose Latest.
    4. Leave the Cluster as is (pointing to the cluster that you are configuring).
    5. Pick a name for Service name.
    6. Set 1 for Number of tasks.
    7. Choose Rolling update for Deployment type.
  9. Choose Next step.
  10. For Configure network,
    1. For Cluster VPC, choose the VPC that you have created previously.
    2. Choose the only public subnet.
  11. For Load balancing, choose None as Load balancer type.
  12. For Auto-assign public IP keep it as ENABLED.
  13. Choose Next step.
  14. Leave options in Set Auto Scaling (optional) to the default : Do not adjust the service’s desired count
  15. Choose Next step and Create Service.
  16. Choose View Service and wait for the task to get to the running state.
  17. Once the task is running, check if the slack bot is responding to messages in DM's or in a channel it is invited to.