Find Jenkins EC2 build nodes that Jenkins has lost track of.
This is deployed as an AWS Lambda function and can be setup to run periodically.
It sends Slack notifications listing instance ids of leaked nodes. Other notification methods can be easily implemented.
Lambda functions are a little difficult to configure. This uses a two-stage process:
- A property file specifies the S3 location of a configuration file. This is included in the uberjar.
- An EDN configuration file found at the location specified above contains other configuration, including Jenkins and notofication credentials.
- Check the code. Your criteria for finding leaked Jenkins node may differ.
- Set your bucket and prefix in resources/lambda.properties.
lein uberjar- Make your config.edn and upload it to
s3://your-bucket/prefix/FUNCTION_NAME/config.edn - Create a Lambda IAM role with at least the following permissions:
ec2:DescribeInstances,s3:GetObjecton yourconfig.ednin S3, plus the AWS Logs permissions. - Create the function:
aws lambda create-function --region REGION --function-name FUNCTION_NAME --zip-file fileb://target/uberjar/jenkins-orphan-nodes-0.1.0-SNAPSHOT-standalone.jar --role ROLE_ARN --runtime java8 --handler jenkins_orphan_nodes.core.LambdaFn --timeout 59 --memory-size 512 - Test the function.
To upload a new version of the code:
lein uberjaraws lambda update-function-code --region REGION --function-name FUNCTION_NAME --zip-file fileb://target/uberjar/jenkins-orphan-nodes-0.1.0-SNAPSHOT-standalone.jar --publish
The code can be tested from the REPL as normal.
The dev.sh script is an example of running the REPL process in a Docker container. I ran into problems with SSL certificate on my Jenkins server. A Clojure process running on my Mac refused to validate my certificate, but on Linux (Lambda, Docker) it worked fine.
Copyright © 2016 Yummly
Distributed under the Eclipse Public License either version 1.0 or (at your option) any later version.