Skip to content

Commit

Permalink
DOM-52659 Provision bucket for flyte (#188)
Browse files Browse the repository at this point in the history 
* provision bucket for flyte

* update readme

* add flyte enabled to more places

* provision separate buckets for flyte metadata and flyte data

* update readme

* no _ in bucket name

* try making flyte_enabled a top level setting

* default to false, remove nullable

* make flyte a top level object instead of a key so it can grow

* update docs

* omit flyte from single node tfvars

* clean up lingering reference to flyte_enabled
  • Loading branch information
@noahjax
noahjax authored Jan 12, 2024
1 parent c33c266 commit 5d81a5b
Show file tree
Hide file tree
Showing 7 changed files with 179 additions and 0 deletions.
1 change: 1 addition & 0 deletions modules/infra/README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -61,6 +61,7 @@
| <a name="input_deploy_id"></a> [deploy\_id](#input\_deploy\_id) | Domino Deployment ID. | `string` | `"domino-eks"` | no |
| <a name="input_domino_cur"></a> [domino\_cur](#input\_domino\_cur) | Determines whether to provision domino cost related infrastructures, ie, long term storage | <pre>object({<br> provision_cost_usage_report = optional(bool, false)<br> })</pre> | `{}` | no |
| <a name="input_eks"></a> [eks](#input\_eks) | creation\_role\_name = Name of the role to import.<br> k8s\_version = EKS cluster k8s version.<br> nodes\_master Grants the nodes role system:master access. NOT recomended<br> kubeconfig = {<br> extra\_args = Optional extra args when generating kubeconfig.<br> path = Fully qualified path name to write the kubeconfig file.<br> }<br> public\_access = {<br> enabled = Enable EKS API public endpoint.<br> cidrs = List of CIDR ranges permitted for accessing the EKS public endpoint.<br> }<br> Custom role maps for aws auth configmap<br> custom\_role\_maps = {<br> rolearn = string<br> username = string<br> groups = list(string)<br> }<br> master\_role\_names = IAM role names to be added as masters in eks.<br> cluster\_addons = EKS cluster addons. vpc-cni is installed separately.<br> vpc\_cni = Configuration for AWS VPC CNI<br> ssm\_log\_group\_name = CloudWatch log group to send the SSM session logs to.<br> identity\_providers = Configuration for IDP(Identity Provider).<br> } | <pre>object({<br> creation_role_name = optional(string, null)<br> k8s_version = optional(string, "1.27")<br> nodes_master = optional(bool, false)<br> kubeconfig = optional(object({<br> extra_args = optional(string, "")<br> path = optional(string, null)<br> }), {})<br> public_access = optional(object({<br> enabled = optional(bool, false)<br> cidrs = optional(list(string), [])<br> }), {})<br> custom_role_maps = optional(list(object({<br> rolearn = string<br> username = string<br> groups = list(string)<br> })), [])<br> master_role_names = optional(list(string), [])<br> cluster_addons = optional(list(string), ["kube-proxy", "coredns"])<br> ssm_log_group_name = optional(string, "session-manager")<br> vpc_cni = optional(object({<br> prefix_delegation = optional(bool)<br> annotate_pod_ip = optional(bool)<br> }))<br> identity_providers = optional(list(object({<br> client_id = string<br> groups_claim = optional(string, null)<br> groups_prefix = optional(string, null)<br> identity_provider_config_name = string<br> issuer_url = optional(string, null)<br> required_claims = optional(string, null)<br> username_claim = optional(string, null)<br> username_prefix = optional(string, null)<br> })), [])<br> })</pre> | `{}` | no |
| <a name="input_flyte"></a> [flyte](#input\_flyte) | enabled = Whether to provision any Flyte related resources | <pre>object({<br> enabled = optional(bool, false)<br> })</pre> | `{}` | no |
| <a name="input_kms"></a> [kms](#input\_kms) | enabled = "Toggle, if set use either the specified KMS key\_id or a Domino-generated one"<br> key\_id = optional(string, null)<br> additional\_policies = "Allows setting additional KMS key policies when using a Domino-generated key" | <pre>object({<br> enabled = optional(bool, true)<br> key_id = optional(string, null)<br> additional_policies = optional(list(string), [])<br> })</pre> | `{}` | no |
| <a name="input_network"></a> [network](#input\_network) | vpc = {<br> id = Existing vpc id, it will bypass creation by this module.<br> subnets = {<br> private = Existing private subnets.<br> public = Existing public subnets.<br> pod = Existing pod subnets.<br> }), {})<br> }), {})<br> network\_bits = {<br> public = Number of network bits to allocate to the public subnet. i.e /27 -> 32 IPs.<br> private = Number of network bits to allocate to the private subnet. i.e /19 -> 8,192 IPs.<br> pod = Number of network bits to allocate to the private subnet. i.e /19 -> 8,192 IPs.<br> }<br> cidrs = {<br> vpc = The IPv4 CIDR block for the VPC.<br> pod = The IPv4 CIDR block for the Pod subnets.<br> }<br> use\_pod\_cidr = Use additional pod CIDR range (ie 100.64.0.0/16) for pod networking. | <pre>object({<br> vpc = optional(object({<br> id = optional(string, null)<br> subnets = optional(object({<br> private = optional(list(string), [])<br> public = optional(list(string), [])<br> pod = optional(list(string), [])<br> }), {})<br> }), {})<br> network_bits = optional(object({<br> public = optional(number, 27)<br> private = optional(number, 19)<br> pod = optional(number, 19)<br> }<br> ), {})<br> cidrs = optional(object({<br> vpc = optional(string, "10.0.0.0/16")<br> pod = optional(string, "100.64.0.0/16")<br> }), {})<br> use_pod_cidr = optional(bool, true)<br> })</pre> | `{}` | no |
| <a name="input_region"></a> [region](#input\_region) | AWS region for the deployment | `string` | n/a | yes |
Expand Down
1 change: 1 addition & 0 deletions modules/infra/main.tf
View file
Original file line number Diff line number Diff line change
Expand Up @@ -29,6 +29,7 @@ module "storage" {
network_info = module.network.info
kms_info = local.kms_info
storage = var.storage
flyte = var.flyte
}

data "aws_ec2_instance_type" "all" {
Expand Down
5 changes: 5 additions & 0 deletions modules/infra/submodules/storage/README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -37,6 +37,8 @@ No modules.
| [aws_s3_bucket.backups](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/s3_bucket) | resource |
| [aws_s3_bucket.blobs](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/s3_bucket) | resource |
| [aws_s3_bucket.costs](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/s3_bucket) | resource |
| [aws_s3_bucket.flyte_data](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/s3_bucket) | resource |
| [aws_s3_bucket.flyte_metadata](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/s3_bucket) | resource |
| [aws_s3_bucket.logs](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/s3_bucket) | resource |
| [aws_s3_bucket.monitoring](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/s3_bucket) | resource |
| [aws_s3_bucket.registry](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/s3_bucket) | resource |
Expand All @@ -56,6 +58,8 @@ No modules.
| [aws_iam_policy_document.blobs](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/data-sources/iam_policy_document) | data source |
| [aws_iam_policy_document.costs](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/data-sources/iam_policy_document) | data source |
| [aws_iam_policy_document.ecr](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/data-sources/iam_policy_document) | data source |
| [aws_iam_policy_document.flyte_data](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/data-sources/iam_policy_document) | data source |
| [aws_iam_policy_document.flyte_metadata](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/data-sources/iam_policy_document) | data source |
| [aws_iam_policy_document.logs](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/data-sources/iam_policy_document) | data source |
| [aws_iam_policy_document.monitoring](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/data-sources/iam_policy_document) | data source |
| [aws_iam_policy_document.registry](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/data-sources/iam_policy_document) | data source |
Expand All @@ -67,6 +71,7 @@ No modules.
| Name | Description | Type | Default | Required |
|------|-------------|------|---------|:--------:|
| <a name="input_deploy_id"></a> [deploy\_id](#input\_deploy\_id) | Domino Deployment ID | `string` | n/a | yes |
| <a name="input_flyte"></a> [flyte](#input\_flyte) | enabled = Whether to provision any Flyte related resources | <pre>object({<br> enabled = optional(bool, false)<br> })</pre> | `{}` | no |
| <a name="input_kms_info"></a> [kms\_info](#input\_kms\_info) | key\_id = KMS key id.<br> key\_arn = KMS key arn.<br> enabled = KMS key is enabled | <pre>object({<br> key_id = string<br> key_arn = string<br> enabled = bool<br> })</pre> | n/a | yes |
| <a name="input_network_info"></a> [network\_info](#input\_network\_info) | id = VPC ID.<br> subnets = {<br> public = List of public Subnets.<br> [{<br> name = Subnet name.<br> subnet\_id = Subnet ud<br> az = Subnet availability\_zone<br> az\_id = Subnet availability\_zone\_id<br> }]<br> private = List of private Subnets.<br> [{<br> name = Subnet name.<br> subnet\_id = Subnet ud<br> az = Subnet availability\_zone<br> az\_id = Subnet availability\_zone\_id<br> }]<br> pod = List of pod Subnets.<br> [{<br> name = Subnet name.<br> subnet\_id = Subnet ud<br> az = Subnet availability\_zone<br> az\_id = Subnet availability\_zone\_id<br> }]<br> } | <pre>object({<br> vpc_id = string<br> subnets = object({<br> public = optional(list(object({<br> name = string<br> subnet_id = string<br> az = string<br> az_id = string<br> })), [])<br> private = list(object({<br> name = string<br> subnet_id = string<br> az = string<br> az_id = string<br> }))<br> pod = optional(list(object({<br> name = string<br> subnet_id = string<br> az = string<br> az_id = string<br> })), [])<br> })<br> })</pre> | n/a | yes |
| <a name="input_storage"></a> [storage](#input\_storage) | storage = {<br> efs = {<br> access\_point\_path = Filesystem path for efs.<br> backup\_vault = {<br> create = Create backup vault for EFS toggle.<br> force\_destroy = Toggle to allow automatic destruction of all backups when destroying.<br> backup = {<br> schedule = Cron-style schedule for EFS backup vault (default: once a day at 12pm).<br> cold\_storage\_after = Move backup data to cold storage after this many days.<br> delete\_after = Delete backup data after this many days.<br> }<br> }<br> }<br> s3 = {<br> force\_destroy\_on\_deletion = Toogle to allow recursive deletion of all objects in the s3 buckets. if 'false' terraform will NOT be able to delete non-empty buckets.<br> }<br> ecr = {<br> force\_destroy\_on\_deletion = Toogle to allow recursive deletion of all objects in the ECR repositories. if 'false' terraform will NOT be able to delete non-empty repositories.<br> }<br> enable\_remote\_backup = Enable tagging required for cross-account backups<br> costs\_enabled = Determines whether to provision domino cost related infrastructures, ie, long term storage<br> }<br> } | <pre>object({<br> efs = optional(object({<br> access_point_path = optional(string)<br> backup_vault = optional(object({<br> create = optional(bool)<br> force_destroy = optional(bool)<br> backup = optional(object({<br> schedule = optional(string)<br> cold_storage_after = optional(number)<br> delete_after = optional(number)<br> }))<br> }))<br> }))<br> s3 = optional(object({<br> force_destroy_on_deletion = optional(bool)<br> }))<br> ecr = optional(object({<br> force_destroy_on_deletion = optional(bool)<br> }))<br> enable_remote_backup = optional(bool)<br> costs_enabled = optional(bool)<br> })</pre> | n/a | yes |
Expand Down
12 changes: 12 additions & 0 deletions modules/infra/submodules/storage/main.tf
View file
Original file line number Diff line number Diff line change
Expand Up @@ -24,6 +24,18 @@ locals {
policy_json = data.aws_iam_policy_document.costs[0].json
arn = aws_s3_bucket.costs[0].arn
} : {}
flyte_metadata = var.flyte.enabled ? {
bucket_name = aws_s3_bucket.flyte_metadata[0].bucket
id = aws_s3_bucket.flyte_metadata[0].id
policy_json = data.aws_iam_policy_document.flyte_metadata[0].json
arn = aws_s3_bucket.flyte_metadata[0].arn
} : {}
flyte_data = var.flyte.enabled ? {
bucket_name = aws_s3_bucket.flyte_data[0].bucket
id = aws_s3_bucket.flyte_data[0].id
policy_json = data.aws_iam_policy_document.flyte_data[0].json
arn = aws_s3_bucket.flyte_data[0].arn
} : {}
logs = {
bucket_name = aws_s3_bucket.logs.bucket
id = aws_s3_bucket.logs.id
Expand Down
138 changes: 138 additions & 0 deletions modules/infra/submodules/storage/s3.tf
View file
Original file line number Diff line number Diff line change
Expand Up @@ -603,3 +603,141 @@ data "aws_iam_policy_document" "costs" {
}
}
}

resource "aws_s3_bucket" "flyte_metadata" {
count = var.flyte.enabled ? 1 : 0
bucket = "${var.deploy_id}-flyte-metadata"
force_destroy = var.storage.s3.force_destroy_on_deletion
object_lock_enabled = false
}

data "aws_iam_policy_document" "flyte_metadata" {
count = var.flyte.enabled ? 1 : 0

statement {
effect = "Deny"

resources = [
"arn:${data.aws_partition.current.partition}:s3:::${aws_s3_bucket.flyte_metadata[0].bucket}",
"arn:${data.aws_partition.current.partition}:s3:::${aws_s3_bucket.flyte_metadata[0].bucket}/*",
]

actions = ["s3:*"]

condition {
test = "Bool"
variable = "aws:SecureTransport"
values = ["false"]
}

principals {
type = "AWS"
identifiers = ["*"]
}
}

statement {
sid = "DenyIncorrectEncryptionHeader"
effect = "Deny"
resources = ["arn:${data.aws_partition.current.partition}:s3:::${aws_s3_bucket.flyte_metadata[0].bucket}/*"]
actions = ["s3:PutObject"]

condition {
test = "StringNotEquals"
variable = "s3:x-amz-server-side-encryption"
values = [local.s3_server_side_encryption]
}

principals {
type = "AWS"
identifiers = ["*"]
}
}

statement {
sid = "DenyUnEncryptedObjectUploads"
effect = "Deny"
resources = ["arn:${data.aws_partition.current.partition}:s3:::${aws_s3_bucket.flyte_metadata[0].bucket}/*"]
actions = ["s3:PutObject"]

condition {
test = "Null"
variable = "s3:x-amz-server-side-encryption"
values = ["true"]
}

principals {
type = "AWS"
identifiers = ["*"]
}
}
}

resource "aws_s3_bucket" "flyte_data" {
count = var.flyte.enabled ? 1 : 0
bucket = "${var.deploy_id}-flyte-data"
force_destroy = var.storage.s3.force_destroy_on_deletion
object_lock_enabled = false
}

data "aws_iam_policy_document" "flyte_data" {
count = var.flyte.enabled ? 1 : 0

statement {
effect = "Deny"

resources = [
"arn:${data.aws_partition.current.partition}:s3:::${aws_s3_bucket.flyte_data[0].bucket}",
"arn:${data.aws_partition.current.partition}:s3:::${aws_s3_bucket.flyte_data[0].bucket}/*",
]

actions = ["s3:*"]

condition {
test = "Bool"
variable = "aws:SecureTransport"
values = ["false"]
}

principals {
type = "AWS"
identifiers = ["*"]
}
}

statement {
sid = "DenyIncorrectEncryptionHeader"
effect = "Deny"
resources = ["arn:${data.aws_partition.current.partition}:s3:::${aws_s3_bucket.flyte_data[0].bucket}/*"]
actions = ["s3:PutObject"]

condition {
test = "StringNotEquals"
variable = "s3:x-amz-server-side-encryption"
values = [local.s3_server_side_encryption]
}

principals {
type = "AWS"
identifiers = ["*"]
}
}

statement {
sid = "DenyUnEncryptedObjectUploads"
effect = "Deny"
resources = ["arn:${data.aws_partition.current.partition}:s3:::${aws_s3_bucket.flyte_data[0].bucket}/*"]
actions = ["s3:PutObject"]

condition {
test = "Null"
variable = "s3:x-amz-server-side-encryption"
values = ["true"]
}

principals {
type = "AWS"
identifiers = ["*"]
}
}
}
11 changes: 11 additions & 0 deletions modules/infra/submodules/storage/variables.tf
View file
Original file line number Diff line number Diff line change
Expand Up @@ -122,3 +122,14 @@ variable "network_info" {
})
})
}

variable "flyte" {
description = <<EOF
enabled = Whether to provision any Flyte related resources
EOF
type = object({
enabled = optional(bool, false)
})

default = {}
}
11 changes: 11 additions & 0 deletions modules/infra/variables.tf
View file
Original file line number Diff line number Diff line change
Expand Up @@ -419,3 +419,14 @@ variable "domino_cur" {

default = {}
}

variable "flyte" {
description = <<EOF
enabled = Whether to provision any Flyte related resources
EOF
type = object({
enabled = optional(bool, false)
})

default = {}
}

0 comments on commit 5d81a5b

Please sign in to comment.