PLAT-6023: Add support for optional pod subnet (#19)

* Also provision "internal" subnet

* Ensure we can cleanly destroy

* Make internal subnets optional

* Run k8s module inside eks module

* Switch CNI to use internal subnets

* Internal subnets optionality in k8s module

* Ensure the -ex is picked up

* Terraform fmt

* terraform fmt

* Update terraform docs

* Never added this

* Explicitly allow egress to internal subnet

Registry etc. not hitting svc's backed by internal subnet.

* Add CIDR to eks cluster security group

TODO: See if changing the ENIConfig to use the node group SG would
fix this. Custom networking example uses cluster SG, but we're not
really geared for that.

* Revert "Add CIDR to eks cluster security group"

This reverts commit 5e04926.

* Revert "Explicitly allow egress to internal subnet"

This reverts commit 753239c.

* Use the node security group

Amazon examples use the cluster group, but that's not actually
right for our infra here.

* Add default to `create_bastion_sg`

* Update terraform-docs

* s/internal_subnets/pod_subnets/

* s/internal_subnets/pod_subnets/

* feedback

* Single set env call

README.md

@@ -85,7 +85,6 @@ aws s3 rb s3://"${AWS_TERRAFORM_REMOTE_STATE_BUCKET}" --force
 |------|--------|---------|
 | <a name="module_bastion"></a> [bastion](#module\_bastion) | ./submodules/bastion | n/a |
 | <a name="module_eks"></a> [eks](#module\_eks) | ./submodules/eks | n/a |
-| <a name="module_k8s_setup"></a> [k8s\_setup](#module\_k8s\_setup) | ./submodules/k8s | n/a |
 | <a name="module_network"></a> [network](#module\_network) | ./submodules/network | n/a |
 | <a name="module_storage"></a> [storage](#module\_storage) | ./submodules/storage | n/a |
 
@@ -99,8 +98,8 @@ aws s3 rb s3://"${AWS_TERRAFORM_REMOTE_STATE_BUCKET}" --force
 | [aws_availability_zones.available](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/data-sources/availability_zones) | data source |
 | [aws_ec2_instance_type_offerings.nodes](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/data-sources/ec2_instance_type_offerings) | data source |
 | [aws_iam_policy_document.route53](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/data-sources/iam_policy_document) | data source |
-| [aws_iam_role.eks_master_roles](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/data-sources/iam_role) | data source |
 | [aws_route53_zone.hosted](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/data-sources/route53_zone) | data source |
+| [aws_subnet.pod](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/data-sources/subnet) | data source |
 | [aws_subnet.private](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/data-sources/subnet) | data source |
 | [aws_subnet.public](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/data-sources/subnet) | data source |
 | [tls_public_key.domino](https://registry.terraform.io/providers/hashicorp/tls/latest/docs/data-sources/public_key) | data source |
@@ -120,6 +119,9 @@ aws s3 rb s3://"${AWS_TERRAFORM_REMOTE_STATE_BUCKET}" --force
 | <a name="input_k8s_version"></a> [k8s\_version](#input\_k8s\_version) | EKS cluster k8s version. | `string` | `"1.24"` | no |
 | <a name="input_kubeconfig_path"></a> [kubeconfig\_path](#input\_kubeconfig\_path) | fully qualified path name to write the kubeconfig file | `string` | `""` | no |
 | <a name="input_number_of_azs"></a> [number\_of\_azs](#input\_number\_of\_azs) | Number of AZ to distribute the deployment, EKS needs at least 2. | `number` | `3` | no |
+| <a name="input_pod_cidr"></a> [pod\_cidr](#input\_pod\_cidr) | The IPv4 CIDR block for the VPC. | `string` | `"100.64.0.0/16"` | no |
+| <a name="input_pod_cidr_network_bits"></a> [pod\_cidr\_network\_bits](#input\_pod\_cidr\_network\_bits) | Number of network bits to allocate to the private subnet. i.e /19 -> 8,192 IPs. | `number` | `19` | no |
+| <a name="input_pod_subnets"></a> [pod\_subnets](#input\_pod\_subnets) | Optional list of pod subnet ids | `list(string)` | `null` | no |
 | <a name="input_private_cidr_network_bits"></a> [private\_cidr\_network\_bits](#input\_private\_cidr\_network\_bits) | Number of network bits to allocate to the private subnet. i.e /19 -> 8,192 IPs. | `number` | `19` | no |
 | <a name="input_private_subnets"></a> [private\_subnets](#input\_private\_subnets) | Optional list of private subnet ids | `list(string)` | `null` | no |
 | <a name="input_public_cidr_network_bits"></a> [public\_cidr\_network\_bits](#input\_public\_cidr\_network\_bits) | Number of network bits to allocate to the public subnet. i.e /27 -> 32 IPs. | `number` | `27` | no |
@@ -129,6 +131,8 @@ aws s3 rb s3://"${AWS_TERRAFORM_REMOTE_STATE_BUCKET}" --force
 | <a name="input_s3_force_destroy_on_deletion"></a> [s3\_force\_destroy\_on\_deletion](#input\_s3\_force\_destroy\_on\_deletion) | Toogle to allow recursive deletion of all objects in the s3 buckets. if 'false' terraform will NOT be able to delete non-empty buckets | `bool` | `false` | no |
 | <a name="input_ssh_pvt_key_path"></a> [ssh\_pvt\_key\_path](#input\_ssh\_pvt\_key\_path) | SSH private key filepath. | `string` | n/a | yes |
 | <a name="input_tags"></a> [tags](#input\_tags) | Deployment tags. | `map(string)` | `{}` | no |
+| <a name="input_update_kubeconfig_extra_args"></a> [update\_kubeconfig\_extra\_args](#input\_update\_kubeconfig\_extra\_args) | Optional extra args when generating kubeconfig | `string` | `""` | no |
+| <a name="input_use_pod_cidr"></a> [use\_pod\_cidr](#input\_use\_pod\_cidr) | Use additional pod CIDR range (ie 100.64.0.0/16) for pod/service networking | `bool` | `true` | no |
 | <a name="input_vpc_id"></a> [vpc\_id](#input\_vpc\_id) | Optional VPC ID, it will bypass creation of such. public\_subnets and private\_subnets are also required. | `string` | `null` | no |
 
 ## Outputs

main.tf

@@ -39,6 +39,11 @@ data "aws_subnet" "private" {
   id    = var.private_subnets[count.index]
 }
 
+data "aws_subnet" "pod" {
+  count = var.vpc_id != null ? length(var.pod_subnets) : 0
+  id    = var.pod_subnets[count.index]
+}
+
 locals {
   # Get zones where ALL instance types are offered(intersection).
   zone_intersection_instance_offerings = setintersection([for k, v in data.aws_ec2_instance_type_offerings.nodes : toset(v.locations)]...)
@@ -89,6 +94,12 @@ locals {
   public_cidr_blocks = slice(local.subnet_cidr_blocks, 0, local.num_of_azs)
   ## Match the private subnet var to the list of cidr blocks
   private_cidr_blocks = slice(local.subnet_cidr_blocks, local.num_of_azs, length(local.subnet_cidr_blocks))
+  ## Determine cidr blocks for pod network
+  base_pod_cidr_network_bits = tonumber(regex("[^/]*$", var.pod_cidr))
+  pod_cidr_blocks = !var.use_pod_cidr ? [] : cidrsubnets(
+    var.pod_cidr,
+    [for n in range(0, local.num_of_azs) : var.pod_cidr_network_bits - local.base_pod_cidr_network_bits]...
+  )
 }
 
 module "network" {
@@ -98,16 +109,20 @@ module "network" {
   deploy_id           = var.deploy_id
   region              = var.region
   cidr                = var.cidr
+  pod_cidr            = var.pod_cidr
+  use_pod_cidr        = var.use_pod_cidr
   availability_zones  = local.azs_to_use
   public_cidrs        = local.public_cidr_blocks
   private_cidrs       = local.private_cidr_blocks
+  pod_cidrs           = local.pod_cidr_blocks
   flow_log_bucket_arn = { arn = module.storage.s3_buckets["monitoring"].arn }
 }
 
 locals {
   vpc_id          = var.vpc_id != null ? var.vpc_id : module.network[0].vpc_id
   public_subnets  = var.vpc_id != null ? [for s in data.aws_subnet.public : { subnet_id = s.id, az = s.availability_zone }] : module.network[0].public_subnets
   private_subnets = var.vpc_id != null ? [for s in data.aws_subnet.private : { subnet_id = s.id, az = s.availability_zone }] : module.network[0].private_subnets
+  pod_subnets     = var.vpc_id != null ? [for s in data.aws_subnet.pod : { subnet_id = s.id, az = s.availability_zone }] : module.network[0].pod_subnets
 }
 
 module "bastion" {
@@ -130,7 +145,8 @@ module "eks" {
   k8s_version                  = var.k8s_version
   vpc_id                       = local.vpc_id
   private_subnets              = local.private_subnets
-  ssh_pvt_key_path             = aws_key_pair.domino.key_name
+  pod_subnets                  = local.pod_subnets
+  ssh_key_pair_name            = aws_key_pair.domino.key_name
   bastion_security_group_id    = try(module.bastion[0].security_group_id, "")
   create_bastion_sg            = var.bastion != null
   kubeconfig_path              = local.kubeconfig_path
@@ -139,28 +155,12 @@ module "eks" {
   node_iam_policies            = [module.storage.s3_policy]
   efs_security_group           = module.storage.efs_security_group
   update_kubeconfig_extra_args = var.update_kubeconfig_extra_args
+  eks_master_role_names        = var.eks_master_role_names
+  ssh_pvt_key_path             = local.ssh_pvt_key_path
+  bastion_user                 = local.bastion_user
+  bastion_public_ip            = try(module.bastion[0].public_ip, "")
 
   depends_on = [
     module.network
   ]
 }
-
-data "aws_iam_role" "eks_master_roles" {
-  for_each = var.bastion != null ? toset(var.eks_master_role_names) : []
-  name     = each.key
-}
-
-module "k8s_setup" {
-  count                = var.bastion != null ? 1 : 0
-  source               = "./submodules/k8s"
-  ssh_pvt_key_path     = local.ssh_pvt_key_path
-  bastion_user         = local.bastion_user
-  bastion_public_ip    = try(module.bastion[0].public_ip, "")
-  eks_node_role_arns   = [for r in module.eks.eks_node_roles : r.arn]
-  eks_master_role_arns = [for r in concat(values(data.aws_iam_role.eks_master_roles), module.eks.eks_master_roles) : r.arn]
-  kubeconfig_path      = local.kubeconfig_path
-  depends_on = [
-    module.eks,
-    module.bastion
-  ]
-}

submodules/bastion/main.tf

@@ -14,7 +14,7 @@ resource "aws_security_group" "bastion" {
 
   lifecycle {
     create_before_destroy = true
-    ignore_changes = [description]
+    ignore_changes        = [description]
   }
 
   tags = {

submodules/eks/README.md

@@ -18,7 +18,9 @@
 
 ## Modules
 
-No modules.
+| Name | Source | Version |
+|------|--------|---------|
+| <a name="module_k8s_setup"></a> [k8s\_setup](#module\_k8s\_setup) | ../k8s | n/a |
 
 ## Resources
 
@@ -27,6 +29,7 @@ No modules.
 | [aws_autoscaling_group_tag.tag](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/autoscaling_group_tag) | resource |
 | [aws_cloudwatch_log_group.eks_cluster](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/cloudwatch_log_group) | resource |
 | [aws_eks_addon.this](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/eks_addon) | resource |
+| [aws_eks_addon.vpc_cni](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/eks_addon) | resource |
 | [aws_eks_cluster.this](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/eks_cluster) | resource |
 | [aws_eks_node_group.node_groups](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/eks_node_group) | resource |
 | [aws_iam_policy.custom_eks_node_policy](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/iam_policy) | resource |
@@ -54,25 +57,32 @@ No modules.
 | [aws_iam_policy_document.eks_nodes](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/data-sources/iam_policy_document) | data source |
 | [aws_iam_policy_document.kms_key](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/data-sources/iam_policy_document) | data source |
 | [aws_iam_policy_document.snapshot](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/data-sources/iam_policy_document) | data source |
+| [aws_iam_role.eks_master_roles](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/data-sources/iam_role) | data source |
 | [aws_partition.current](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/data-sources/partition) | data source |
 
 ## Inputs
 
 | Name | Description | Type | Default | Required |
 |------|-------------|------|---------|:--------:|
 | <a name="input_additional_node_groups"></a> [additional\_node\_groups](#input\_additional\_node\_groups) | Additional EKS managed node groups definition. | <pre>map(object({<br>    ami                  = optional(string)<br>    bootstrap_extra_args = optional(string, "")<br>    instance_types       = list(string)<br>    spot                 = optional(bool, false)<br>    min_per_az           = number<br>    max_per_az           = number<br>    desired_per_az       = number<br>    labels               = map(string)<br>    taints               = optional(list(object({ key = string, value = optional(string), effect = string })), [])<br>    tags                 = optional(map(string), {})<br>    volume = object({<br>      size = string<br>      type = string<br>    })<br>  }))</pre> | `{}` | no |
+| <a name="input_bastion_public_ip"></a> [bastion\_public\_ip](#input\_bastion\_public\_ip) | Public IP of bastion instance | `string` | `""` | no |
 | <a name="input_bastion_security_group_id"></a> [bastion\_security\_group\_id](#input\_bastion\_security\_group\_id) | Bastion security group id. | `string` | `""` | no |
-| <a name="input_create_bastion_sg"></a> [create\_bastion\_sg](#input\_create\_bastion\_sg) | Create bastion access rules toggle. | `bool` | n/a | yes |
+| <a name="input_bastion_user"></a> [bastion\_user](#input\_bastion\_user) | Username for bastion instance | `string` | `""` | no |
+| <a name="input_create_bastion_sg"></a> [create\_bastion\_sg](#input\_create\_bastion\_sg) | Create bastion access rules toggle. | `bool` | `false` | no |
 | <a name="input_default_node_groups"></a> [default\_node\_groups](#input\_default\_node\_groups) | EKS managed node groups definition. | <pre>object(<br>    {<br>      compute = object(<br>        {<br>          ami                  = optional(string)<br>          bootstrap_extra_args = optional(string, "")<br>          instance_types       = optional(list(string), ["m5.2xlarge"])<br>          spot                 = optional(bool, false)<br>          min_per_az           = optional(number, 0)<br>          max_per_az           = optional(number, 10)<br>          desired_per_az       = optional(number, 1)<br>          labels = optional(map(string), {<br>            "dominodatalab.com/node-pool" = "default"<br>          })<br>          taints = optional(list(object({ key = string, value = optional(string), effect = string })), [])<br>          tags   = optional(map(string), {})<br>          volume = optional(object(<br>            {<br>              size = optional(number, 100)<br>              type = optional(string, "gp3")<br>            }),<br>            {<br>              size = 100<br>              type = "gp3"<br>            }<br>          )<br>      }),<br>      platform = object(<br>        {<br>          ami                  = optional(string)<br>          bootstrap_extra_args = optional(string, "")<br>          instance_types       = optional(list(string), ["m5.4xlarge"])<br>          spot                 = optional(bool, false)<br>          min_per_az           = optional(number, 0)<br>          max_per_az           = optional(number, 10)<br>          desired_per_az       = optional(number, 1)<br>          labels = optional(map(string), {<br>            "dominodatalab.com/node-pool" = "platform"<br>          })<br>          taints = optional(list(object({ key = string, value = optional(string), effect = string })), [])<br>          tags   = optional(map(string), {})<br>          volume = optional(object(<br>            {<br>              size = optional(number, 100)<br>              type = optional(string, "gp3")<br>            }),<br>            {<br>              size = 100<br>              type = "gp3"<br>            }<br>          )<br>      }),<br>      gpu = object(<br>        {<br>          ami                  = optional(string)<br>          bootstrap_extra_args = optional(string, "")<br>          instance_types       = optional(list(string), ["g4dn.xlarge"])<br>          spot                 = optional(bool, false)<br>          min_per_az           = optional(number, 0)<br>          max_per_az           = optional(number, 10)<br>          desired_per_az       = optional(number, 0)<br>          labels = optional(map(string), {<br>            "dominodatalab.com/node-pool" = "default-gpu"<br>            "nvidia.com/gpu"              = true<br>          })<br>          taints = optional(list(object({ key = string, value = optional(string), effect = string })), [])<br>          tags   = optional(map(string), {})<br>          volume = optional(object(<br>            {<br>              size = optional(number, 100)<br>              type = optional(string, "gp3")<br>            }),<br>            {<br>              size = 100<br>              type = "gp3"<br>            }<br>          )<br>      })<br>  })</pre> | <pre>{<br>  "compute": {},<br>  "gpu": {},<br>  "platform": {}<br>}</pre> | no |
 | <a name="input_deploy_id"></a> [deploy\_id](#input\_deploy\_id) | Domino Deployment ID | `string` | n/a | yes |
 | <a name="input_efs_security_group"></a> [efs\_security\_group](#input\_efs\_security\_group) | Security Group ID for EFS | `string` | n/a | yes |
-| <a name="input_eks_cluster_addons"></a> [eks\_cluster\_addons](#input\_eks\_cluster\_addons) | EKS cluster addons. | `list(string)` | <pre>[<br>  "vpc-cni",<br>  "kube-proxy",<br>  "coredns"<br>]</pre> | no |
+| <a name="input_eks_cluster_addons"></a> [eks\_cluster\_addons](#input\_eks\_cluster\_addons) | EKS cluster addons. vpc-cni is installed separately. | `list(string)` | <pre>[<br>  "kube-proxy",<br>  "coredns"<br>]</pre> | no |
+| <a name="input_eks_master_role_names"></a> [eks\_master\_role\_names](#input\_eks\_master\_role\_names) | IAM role names to be added as masters in eks | `list(string)` | `[]` | no |
 | <a name="input_k8s_version"></a> [k8s\_version](#input\_k8s\_version) | EKS cluster k8s version. | `string` | n/a | yes |
 | <a name="input_kubeconfig_path"></a> [kubeconfig\_path](#input\_kubeconfig\_path) | Kubeconfig file path. | `string` | `"kubeconfig"` | no |
 | <a name="input_node_iam_policies"></a> [node\_iam\_policies](#input\_node\_iam\_policies) | Additional IAM Policy Arns for Nodes | `list(string)` | n/a | yes |
+| <a name="input_pod_subnets"></a> [pod\_subnets](#input\_pod\_subnets) | List of POD subnets IDs and AZ | `list(object({ subnet_id = string, az = string }))` | n/a | yes |
 | <a name="input_private_subnets"></a> [private\_subnets](#input\_private\_subnets) | List of Private subnets IDs and AZ | `list(object({ subnet_id = string, az = string }))` | n/a | yes |
 | <a name="input_region"></a> [region](#input\_region) | AWS region for the deployment | `string` | n/a | yes |
-| <a name="input_ssh_pvt_key_path"></a> [ssh\_pvt\_key\_path](#input\_ssh\_pvt\_key\_path) | SSH private key filepath. | `string` | n/a | yes |
+| <a name="input_ssh_key_pair_name"></a> [ssh\_key\_pair\_name](#input\_ssh\_key\_pair\_name) | SSH key pair name. | `string` | n/a | yes |
+| <a name="input_ssh_pvt_key_path"></a> [ssh\_pvt\_key\_path](#input\_ssh\_pvt\_key\_path) | Path to SSH private key | `string` | `""` | no |
+| <a name="input_update_kubeconfig_extra_args"></a> [update\_kubeconfig\_extra\_args](#input\_update\_kubeconfig\_extra\_args) | Optional extra args when generating kubeconfig | `string` | `""` | no |
 | <a name="input_vpc_id"></a> [vpc\_id](#input\_vpc\_id) | VPC ID. | `string` | n/a | yes |
 
 ## Outputs

submodules/eks/cluster.tf

@@ -46,7 +46,7 @@ resource "aws_security_group" "eks_cluster" {
 
   lifecycle {
     create_before_destroy = true
-    ignore_changes = [description, name]
+    ignore_changes        = [description, name]
   }
   tags = {
     "Name"                                            = "${local.eks_cluster_name}-eks-cluster"
@@ -105,6 +105,12 @@ resource "aws_eks_cluster" "this" {
   ]
 }
 
+resource "aws_eks_addon" "vpc_cni" {
+  cluster_name      = aws_eks_cluster.this.name
+  resolve_conflicts = "OVERWRITE"
+  addon_name        = "vpc-cni"
+}
+
 resource "aws_eks_addon" "this" {
   for_each          = toset(var.eks_cluster_addons)
   cluster_name      = aws_eks_cluster.this.name

submodules/eks/iam.tf

@@ -15,7 +15,7 @@ resource "aws_iam_role" "eks_cluster" {
   name               = "${var.deploy_id}-eks"
   assume_role_policy = data.aws_iam_policy_document.eks_cluster.json
   lifecycle {
-      ignore_changes = [name]
+    ignore_changes = [name]
   }
 }
 

submodules/eks/k8s.tf

@@ -0,0 +1,20 @@
+data "aws_iam_role" "eks_master_roles" {
+  for_each = var.create_bastion_sg ? toset(var.eks_master_role_names) : []
+  name     = each.key
+}
+
+module "k8s_setup" {
+  count                = var.create_bastion_sg ? 1 : 0
+  source               = "../k8s"
+  ssh_pvt_key_path     = var.ssh_pvt_key_path
+  bastion_user         = var.bastion_user
+  bastion_public_ip    = try(var.bastion_public_ip, "")
+  eks_node_role_arns   = [aws_iam_role.eks_nodes.arn]
+  eks_master_role_arns = [for r in concat(values(data.aws_iam_role.eks_master_roles), [aws_iam_role.eks_cluster]) : r.arn]
+  kubeconfig_path      = var.kubeconfig_path
+
+  security_group_id = aws_security_group.eks_nodes.id
+  pod_subnets       = var.pod_subnets
+
+  depends_on = [aws_eks_addon.vpc_cni, null_resource.kubeconfig]
+}