-
Notifications
You must be signed in to change notification settings - Fork 89
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Add infrastructure for on-demand ARM64 runners on AWS (#1569)
* Add infrastructure for on-demand ARM64 runners on AWS With this change, ARM64 release artifacts will be built automatically by a GitHub workflow. Since GitHub doesn't offer hosted runners running on ARM64, we're spinning up an EC2 spot instance on demand and run the jobs building ARM64 artifacts there. As a fun side note, the Terraform infrastructure code is written entirely in Nickel. * Remove unused `update-github` script * Address comments from code review * Address comments from code review
- Loading branch information
Showing
14 changed files
with
877 additions
and
12 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,3 @@ | ||
| .terraform* | ||
| build | ||
| main.tf.json |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,56 @@ | ||
| # GitHub Runner Infrastructure | ||
|
|
||
| If you make any changes to the infrastructure code in this directory, you will | ||
| have to redeploy it. Do the following: | ||
|
|
||
| 1. Make sure you're logged into AWS. You can check using `awscli2`: | ||
|
|
||
| ```console | ||
| ❯ nix run nixpkgs#awscli2 -- sts get-caller-identity | ||
| { | ||
| # CENSORED | ||
| } | ||
| ``` | ||
|
|
||
| If this fails, log in with AWS SSO credentials, following [their guide][aws-sso-guide]. | ||
|
|
||
| 2. Make sure you're logged into GitHub. You can check using `gh`: | ||
|
|
||
| ```console | ||
| ❯ nix run github:nixos/nixpkgs#gh -- auth status | ||
| github.com | ||
| # CENSORED | ||
| ✓ Token scopes: gist, read:org, repo | ||
| ``` | ||
|
|
||
| If this fails, log in using `nix run nixpkgs#gh -- auth login` and follow | ||
| the instructions. | ||
|
|
||
| 3. Update the infrastructure using | ||
|
|
||
| ```console | ||
| nix develop ..#infra -c update-infra | ||
| ``` | ||
|
|
||
| ## Architecture | ||
|
|
||
| The code in this subdirectory provisions AWS infrastucture for starting an | ||
| ARM64 GitHub Actions runner on demand. The workflow for producing ARM64 release | ||
| artifacts is as follows: | ||
|
|
||
| - the release workflow is triggered automatically when a release is created or | ||
| manually for testing | ||
| - the workflow requests a runner registration token `$TOKEN` from the GitHub | ||
| API. For this, it needs a personal access token with `repo` scope for the Nickel | ||
| repository. | ||
| - the workflow invokes the `$EC2_START` AWS Lambda and provides `$TOKEN` as | ||
| input | ||
| - the AWS Lambda stores `$TOKEN` as a parameter in the AWS SSM and requests an | ||
| appropriate EC2 spot instance | ||
| - the spot instance boots up, retrieves `$TOKEN` from AWS SSM and starts a | ||
| GitHub Actions runner | ||
| - GitHub Actions schedules the ARM64 jobs on the spot instance | ||
| - when the jobs building the release artifact have finished, the workflow | ||
| invokes the `$EC2_STOP` AWS Lambda which terminates the EC2 instance | ||
|
|
||
| [aws-sso-guide]: https://docs.aws.amazon.com/cli/latest/userguide/sso-configure-profile-token.html |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,49 @@ | ||
| { | ||
| naming_prefix | String, | ||
| github = { | ||
| owner | String, | ||
| repo | String, | ||
| ec2_role = "${resource.aws_iam_role.invoke_lambda_role.arn}", | ||
| }, | ||
| lambda.invoke_policy | String, | ||
| config = { | ||
| resource.aws_iam_openid_connect_provider.github_oidc = { | ||
| url = "https://token.actions.githubusercontent.com", | ||
| client_id_list = [ | ||
| "sts.amazonaws.com", | ||
| ], | ||
| thumbprint_list | ||
| | doc m%" | ||
| Thumbprints are provided by GitHub, see | ||
| [https://github.blog/changelog/2023-06-27-github-actions-update-on-oidc-integration-with-aws/] | ||
| This should be kept sorted to prevent apparent Terraform drift | ||
| "% | ||
| = [ | ||
| "1c58a3a8518e8759bf075b76b750d4f2df264fcd", | ||
| "6938fd4d98bab03faadb97b34396831e3780aea1", | ||
| ], | ||
| }, | ||
|
|
||
| resource.aws_iam_role.invoke_lambda_role = { | ||
| name = "%{naming_prefix}-invoke-lambda-role", | ||
| managed_policy_arns = [lambda.invoke_policy], | ||
| assume_role_policy = | ||
| std.serialize | ||
| 'Json | ||
| { | ||
| Version = "2012-10-17", | ||
| Statement = [ | ||
| { | ||
| Principal.Federated = "${resource.aws_iam_openid_connect_provider.github_oidc.id}", | ||
| Action = "sts:AssumeRoleWithWebIdentity", | ||
| Condition = { | ||
| StringLike."token.actions.githubusercontent.com:sub" = "repo:%{github.owner}/%{github.repo}:ref:refs/tags/*", | ||
| StringEquals."token.actions.githubusercontent.com:aud" = "sts.amazonaws.com", | ||
| }, | ||
| Effect = "Allow", | ||
| } | ||
| ], | ||
| }, | ||
| }, | ||
| } | ||
| } |
Oops, something went wrong.