Automatically update IAM roles with defined policies

This project creates Lambda function that automatically add required AWS Identity and Access Management (IAM) policies to current Amazon Elastic Compute Cloud (Amazon EC2) instance profiles or associate a profile to EC2 instances without a profile associated.
You can use it as a Lambda function scheduled at a recurrent time using EventBridge Rule or manually running the python code when required.

It is a best practice to manage your EC2 instances using AWS Systems Manager Session Manager and get EC2 instance's memory usage metric via CloudWatch Agent.
In order to allow SSM Agent and CloudWatch Agent to communicate with their respective endpoint, EC2 instance needs to have some permissions.

Overview

The CloudFormation template cloudformation/template.yml creates a stack with the following resources:

1. AWS Lambda function. The function's code is in lambda/add_policy_to_ec2_instance.py and is written in Python compatible with version 3.11.
2. Lambda function's execution role.
3. Event Bridge role to execute Lambda at a regular time.

                          +-----------------+
                          | Lambda          |
                          | Execution Role  |
                          +--------+--------+
                                   |
                                   |
+--------------------+    +--------+--------+      +-------------------+
|Event Bridge Rule   +--->+ Lambda function +------>+IAM Role          |
+--------------------+    +--------+--------+      +-------------------+
                                   |
                                   v
                          +--------+--------+
                          | CloudWatch Logs |
                          +-----------------+

NOTE ABOUT REGIONS DEPLOY
There is no reason to deploy this solution twice inside the same region.
If you have a reason for doing it, please open an issue and let's talk about it. IAM Roles are global, but EC2 instances where roles are associated are regional, so you need to deploy this solution on each region where you want to ensure EC2 instances will have the correct policies attached on it.

Lambda configuration

By default, Lambda will add the following policies to any role that is associated with an EC2 instance. If policy is already attached to the role, it will do nothing.

If you need to add more policies, change the list associated with variable POLICIES_TO_ADD.

POLICIES_TO_ADD = [
    'arn:aws:iam::aws:policy/CloudWatchAgentServerPolicy',
    'arn:aws:iam::aws:policy/AmazonSSMManagedInstanceCore'
    ]

NOTE
It doesn't work with in-line policies!

If you have a multi-account environment with AWS Organizations, you should use Default Host Management Configuration (DHMC) from AWS Systems Manager.
https://aws.amazon.com/about-aws/whats-new/2023/10/enable-aws-systems-manager-ec2-instances-organization/

You can also set Lambda environment variable called INSTANCE_PROFILE_NAME with the name of the IAM instance profile to associate with EC2 instance. It will only associate this instance profile if the instance doesn't have any profile associated yet.

Also, if this environment variable is empty or not found, it will do nothing!

You can list your current IAM instance profiles using AWS CLI command below:

aws iam list-instance-profiles --query 'sort_by(InstanceProfiles, &InstanceProfileName)[].InstanceProfileName'

Setup

These are the overall steps to deploy:

Setup using CloudFormation

1. Validate CloudFormation template file.
2. Create the CloudFormation stack.
3. Package the Lambda code into a .zip file.
4. Update Lambda function with the packaged code.

Setup using Terraform

1. Initialize Terraform state
2. Validate Terraform template.
3. Apply Terraform template.

After setup

1. Trigger a test Lambda invocation.
2. Clean-up

Setup using CloudFormation

To simplify setup and deployment, assign the values to the following variables. Replace the values according to your deployment options.

export AWS_REGION="sa-east-1"
export CFN_STACK_NAME="add-policy-to-ec2-instance"

IMPORTANT: Please, use AWS CLI v2

1. Validate CloudFormation template

Ensure the CloudFormation template is valid before use it.

aws cloudformation validate-template --template-body file://cloudformation/template.yml

2. Create CloudFormation stack

At this point it will create Lambda function with a dummy code.
You will update it later.

aws cloudformation create-stack --stack-name "${CFN_STACK_NAME}" \
  --capabilities CAPABILITY_IAM \
  --template-body file://cloudformation/template.yml && {
    ### Wait for stack to be created
    aws cloudformation wait stack-create-complete --stack-name "${CFN_STACK_NAME}"
}

If the stack creation fails, troubleshoot by reviewing the stack events. The typical failure reasons are insufficient IAM permissions.

3. Create the packaged code

zip --junk-paths lambda.zip lambda/add_policy_to_ec2_instance.py

4. Update lambda package code

FUNCTION_NAME=$(aws cloudformation describe-stack-resources --stack-name "${CFN_STACK_NAME}" --query "StackResources[?LogicalResourceId=='LambdaFunction'].PhysicalResourceId" --output text)
aws lambda update-function-code --function-name "${FUNCTION_NAME}" --zip-file fileb://lambda.zip --publish

Setup using Terraform

Terraform template uses the following providers:

• aws
• archive

IMPORTANT: Please, use Terraform version 1.5.2 or higher

1. Initialize Terraform state

cd terraform/
terraform init

2. Validate Terraform template

Ensure Terraform template is valid before use it.

terraform validate

3. Apply Terraform template

terraform apply

After setup

1a. Trigger a test Lambda invocation with the AWS CLI

After the stack is created, AWS resources are not updated until the schedule time. To test the function and update AWS resources with the policies defined for the first time, do a test invocation with the AWS CLI command below:

CloudFormation

aws lambda invoke --function-name "${FUNCTION_NAME}" lambda_return.json

Terraform

FUNCTION_NAME=$(terraform output | grep 'lambda_name' | cut -d ' ' -f 3 | tr -d '"')
aws lambda invoke --function-name "${FUNCTION_NAME}" lambda_return.json

After successful invocation, you should receive the response below with no errors.

{
    "StatusCode": 200,
    "ExecutedVersion": "$LATEST"
}

The content of the lambda_return.json will list all roles updated and instances with profile added.

1b. Trigger a test Lambda invocation with the AWS Console

Alternatively, you can invoke the test event in the AWS Lambda console with sample event below.

{ }

2. Clean-up

Remove the temporary files, remove CloudFormation stack and destroy Terraform resources.

CloudFormation

rm lambda.zip
rm lambda_return.json
aws cloudformation delete-stack --stack-name "${CFN_STACK_NAME}"
unset AWS_REGION
unset CFN_STACK_NAME

Terraform

rm lambda_return.json
terraform destroy

ATTENTION
When you remove CloudFormation stack, or destroy Terraform resources, it will NOT remove policies from the roles updated by this solution.
If you want to remove it, you need to do it manually.

Lambda function customization

After the stack is created, you can customize the Lambda function's execution log level by editing the function's environment variables.

• LOG_LEVEL: Optional. Set log level to increase or reduce verbosity. The default value is INFO. Possible values are:
  • CRITICAL
  • ERROR
  • WARNING
  • INFO
  • DEBUG

Security

See CONTRIBUTING for more information.

License

This library is licensed under the MIT-0 License. See the LICENSE file.