Neptune Module Enhancements

terraform/modules/neptune-cluster/README.md

@@ -11,11 +11,7 @@ No requirements.
 
 ## Modules
 
-| Name | Source | Version |
-|------|--------|---------|
-| <a name="module_cluster_parameters"></a> [cluster\_parameters](#module\_cluster\_parameters) | git::https://github.com/CBIIT/datacommons-devops.git//terraform/modules/neptune-cluster-parameter-group | Neptune |
-| <a name="module_instance_parameters"></a> [instance\_parameters](#module\_instance\_parameters) | git::https://github.com/CBIIT/datacommons-devops.git//terraform/modules/neptune-instance-parameter-group | Neptune |
-| <a name="module_neptune_instance"></a> [neptune\_instance](#module\_neptune\_instance) | git::https://github.com/CBIIT/datacommons-devops.git//terraform/modules/neptune-instance | Neptune |
+No modules.
 
 ## Resources
 
@@ -25,6 +21,9 @@ No requirements.
 | [aws_kms_key.this](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/kms_key) | resource |
 | [aws_kms_key_policy.this](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/kms_key_policy) | resource |
 | [aws_neptune_cluster.this](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/neptune_cluster) | resource |
+| [aws_neptune_cluster_instance.this](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/neptune_cluster_instance) | resource |
+| [aws_neptune_cluster_parameter_group.this](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/neptune_cluster_parameter_group) | resource |
+| [aws_neptune_parameter_group.this](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/neptune_parameter_group) | resource |
 | [aws_neptune_subnet_group.this](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/neptune_subnet_group) | resource |
 | [aws_caller_identity.current](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/data-sources/caller_identity) | data source |
 | [aws_iam_policy_document.kms](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/data-sources/iam_policy_document) | data source |
@@ -39,26 +38,33 @@ No requirements.
 | <a name="input_auto_minor_version_upgrade"></a> [auto\_minor\_version\_upgrade](#input\_auto\_minor\_version\_upgrade) | indicates that minor engine upgrades are applied automatically to the instance during the maintenance window | `bool` | `true` | no |
 | <a name="input_backup_retention_period"></a> [backup\_retention\_period](#input\_backup\_retention\_period) | number of days to retain backups for | `string` | `1` | no |
 | <a name="input_copy_tags_to_snapshot"></a> [copy\_tags\_to\_snapshot](#input\_copy\_tags\_to\_snapshot) | whether to copy tags to snapshots | `bool` | `true` | no |
+| <a name="input_create_kms_key"></a> [create\_kms\_key](#input\_create\_kms\_key) | whether to create the kms key that encrypts the cluster and instance(s) | `bool` | `true` | no |
+| <a name="input_create_parameter_groups"></a> [create\_parameter\_groups](#input\_create\_parameter\_groups) | whether to create parameter groups for the cluster and instance(s) | `bool` | `false` | no |
 | <a name="input_database_subnet_ids"></a> [database\_subnet\_ids](#input\_database\_subnet\_ids) | the list of subnet IDs to associate with the cluster | `set(string)` | n/a | yes |
 | <a name="input_deletion_protection"></a> [deletion\_protection](#input\_deletion\_protection) | whether to enable deletion protection | `bool` | `true` | no |
+| <a name="input_enable_audit_log"></a> [enable\_audit\_log](#input\_enable\_audit\_log) | whether to enable audit logs at the cluster level | `bool` | `true` | no |
 | <a name="input_enable_caching"></a> [enable\_caching](#input\_enable\_caching) | whether to enable caching for the cluster | `bool` | `false` | no |
 | <a name="input_enable_cloudwatch_logs_exports"></a> [enable\_cloudwatch\_logs\_exports](#input\_enable\_cloudwatch\_logs\_exports) | list of log types to export to cloudwatch | `list(string)` | <pre>[<br>  "audit"<br>]</pre> | no |
+| <a name="input_enable_result_cache"></a> [enable\_result\_cache](#input\_enable\_result\_cache) | whether to enable the result cache for the instances in the cluster | `bool` | `false` | no |
 | <a name="input_enable_serverless"></a> [enable\_serverless](#input\_enable\_serverless) | whether to enable serverless mode for the cluster | `bool` | `true` | no |
+| <a name="input_enable_slow_query_log"></a> [enable\_slow\_query\_log](#input\_enable\_slow\_query\_log) | the log level for slow queries applied at the cluster-level - either 'info', 'debug', or 'disable' | `string` | `"info"` | no |
 | <a name="input_engine"></a> [engine](#input\_engine) | the name of the database engine to be used for this instance | `string` | `"neptune"` | no |
-| <a name="input_engine_version"></a> [engine\_version](#input\_engine\_version) | the version of the database engine to use | `string` | `"1.2.1.0"` | no |
+| <a name="input_engine_version"></a> [engine\_version](#input\_engine\_version) | the version of the database engine to use | `string` | `"1.3.1.0"` | no |
 | <a name="input_final_snapshot_identifier"></a> [final\_snapshot\_identifier](#input\_final\_snapshot\_identifier) | the name of the final snapshot to be created immediately before deleting the cluster | `string` | `null` | no |
 | <a name="input_iam_database_authentication_enabled"></a> [iam\_database\_authentication\_enabled](#input\_iam\_database\_authentication\_enabled) | whether to enable IAM database authentication for the cluster | `bool` | `false` | no |
 | <a name="input_iam_roles"></a> [iam\_roles](#input\_iam\_roles) | the list of IAM roles to associate with the cluster | `set(string)` | `[]` | no |
 | <a name="input_instance_class"></a> [instance\_class](#input\_instance\_class) | the instance class to use (i.e., db.r5.large) - only required when serverless is not enabled | `string` | `"db.r5.large"` | no |
 | <a name="input_max_capacity"></a> [max\_capacity](#input\_max\_capacity) | the maximum capacity for the cluster in neptune capacity units when serverless is enabled | `number` | `128` | no |
 | <a name="input_min_capacity"></a> [min\_capacity](#input\_min\_capacity) | the minimum capacity for the cluster in neptune capacity units when serverless is enabled | `number` | `2` | no |
+| <a name="input_parameter_group_family"></a> [parameter\_group\_family](#input\_parameter\_group\_family) | the family of the neptune cluster parameter group (i.e. neptune1.3) | `string` | `"neptune1.3"` | no |
 | <a name="input_port"></a> [port](#input\_port) | the port on which the DB accepts connections | `number` | `8182` | no |
 | <a name="input_preferred_backup_window"></a> [preferred\_backup\_window](#input\_preferred\_backup\_window) | the daily time range during which automated backups are created if automated backups are enabled | `string` | `"02:00-04:00"` | no |
 | <a name="input_preferred_maintenance_window"></a> [preferred\_maintenance\_window](#input\_preferred\_maintenance\_window) | the weekly time range during which system maintenance can occur, in (UTC) | `string` | `"sun:05:00-sun:09:00"` | no |
-| <a name="input_query_timeout"></a> [query\_timeout](#input\_query\_timeout) | time in milliseconds that a query can run before it is terminated by the cluster | `string` | `"120000"` | no |
+| <a name="input_query_timeout"></a> [query\_timeout](#input\_query\_timeout) | time in milliseconds that a query can run before it is terminated by the cluster | `string` | `"60000"` | no |
 | <a name="input_replication_source_identifier"></a> [replication\_source\_identifier](#input\_replication\_source\_identifier) | the ARN of the source Neptune instance if this Neptune instance is a read replica | `string` | `null` | no |
 | <a name="input_resource_prefix"></a> [resource\_prefix](#input\_resource\_prefix) | the prefix to add when creating resources | `string` | n/a | yes |
 | <a name="input_skip_final_snapshot"></a> [skip\_final\_snapshot](#input\_skip\_final\_snapshot) | whether to skip the creation of a final snapshot before deleting the cluster | `bool` | `true` | no |
+| <a name="input_slow_query_log_threshold"></a> [slow\_query\_log\_threshold](#input\_slow\_query\_log\_threshold) | the threshold in milliseconds for slow queries applied at the cluster level | `number` | `5000` | no |
 | <a name="input_snapshot_identifier"></a> [snapshot\_identifier](#input\_snapshot\_identifier) | the name of an existing snapshot from which to create this cluster | `string` | `null` | no |
 | <a name="input_vpc_security_group_ids"></a> [vpc\_security\_group\_ids](#input\_vpc\_security\_group\_ids) | the list of security group IDs to associate with the cluster | `set(string)` | n/a | yes |
 

terraform/modules/neptune-cluster/data.tf

@@ -4,6 +4,7 @@ data "aws_caller_identity" "current" {
 data "aws_region" "current" {}
 
 data "aws_iam_policy_document" "kms" {
+  count = var.create_kms_key ? 1 : 0
 
   statement {
     effect = "Allow"
@@ -52,8 +53,8 @@ data "aws_iam_policy_document" "kms" {
       "kms:Verify"
     ]
     resources = [
-      aws_kms_key.this.arn,
-      aws_kms_alias.this.arn
+      aws_kms_key.this[0].arn,
+      aws_kms_alias.this[0].arn
     ]
   }
 
@@ -78,7 +79,7 @@ data "aws_iam_policy_document" "kms" {
       "kms:DescribeKey"
     ]
     resources = [
-      aws_kms_key.this.arn
+      aws_kms_key.this[0].arn
     ]
     condition {
       test     = "StringEquals"

terraform/modules/neptune-cluster/locals.tf

@@ -0,0 +1,9 @@
+locals {
+  create_parameter_groups = var.enable_serverless ? false : var.create_parameter_groups
+
+  # outputs:
+  kms_key_id     = var.create_kms_key ? aws_kms_key.this[0].key_id : "KMS was created by AWS"
+  kms_alias_arn  = var.create_kms_key ? aws_kms_alias.this[0].arn : "KMS was created by AWS"
+  kms_alias_id   = var.create_kms_key ? aws_kms_alias.this[0].id : "KMS was created by AWS"
+  kms_alias_name = var.create_kms_key ? aws_kms_alias.this[0].name : "KMS was created by AWS"
+}

terraform/modules/neptune-cluster/main.tf

@@ -11,9 +11,9 @@ resource "aws_neptune_cluster" "this" {
   final_snapshot_identifier            = var.final_snapshot_identifier
   iam_roles                            = var.iam_roles
   iam_database_authentication_enabled  = var.iam_database_authentication_enabled
-  kms_key_arn                          = aws_kms_key.this.arn
+  kms_key_arn                          = var.create_kms_key ? aws_kms_alias.this[0].arn : null
   neptune_subnet_group_name            = aws_neptune_subnet_group.this.name
-  neptune_cluster_parameter_group_name = var.enable_serverless ? "default.neptune1.2" : module.cluster_parameters[0].name
+  neptune_cluster_parameter_group_name = var.enable_serverless ? "default.neptune1.3" : aws_neptune_cluster_parameter_group.this[0].name
   preferred_backup_window              = var.preferred_backup_window
   preferred_maintenance_window         = var.preferred_maintenance_window
   port                                 = var.port
@@ -31,55 +31,91 @@ resource "aws_neptune_cluster" "this" {
       min_capacity = var.min_capacity
     }
   }
+
+  depends_on = [aws_kms_key.this, aws_kms_alias.this]
 }
 
-module "cluster_parameters" {
-  count  = var.enable_serverless ? 0 : 1
-  source = "git::https://github.com/CBIIT/datacommons-devops.git//terraform/modules/neptune-cluster-parameter-group?ref=Neptune"
+resource "aws_neptune_cluster_parameter_group" "this" {
+  count = local.create_parameter_groups ? 1 : 0
 
-  resource_prefix  = var.resource_prefix
-  enable_audit_log = var.enable_cloudwatch_logs_exports == ["audit"] ? true : false
-}
+  name        = "${var.resource_prefix}-neptune-cluster-params"
+  family      = var.parameter_group_family
+  description = "${var.resource_prefix} neptune cluster-level parameter group"
+
+  parameter {
+    name  = "neptune_enable_audit_log"
+    value = var.enable_audit_log ? "1" : "0"
+  }
 
-module "instance_parameters" {
-  count  = var.enable_serverless ? 0 : 1
-  source = "git::https://github.com/CBIIT/datacommons-devops.git//terraform/modules/neptune-instance-parameter-group?ref=Neptune"
+  parameter {
+    name  = "neptune_enable_slow_query_log"
+    value = var.enable_slow_query_log
+  }
 
-  resource_prefix = var.resource_prefix
-  enable_caching  = var.enable_serverless ? false : var.enable_caching
-  query_timeout   = var.query_timeout
-}
+  parameter {
+    name  = "neptune_slow_query_log_threshold"
+    value = var.slow_query_log_threshold
+  }
 
-module "neptune_instance" {
-  source = "git::https://github.com/CBIIT/datacommons-devops.git//terraform/modules/neptune-instance?ref=Neptune"
+  parameter {
+    name  = "neptune_query_timeout"
+    value = var.query_timeout
+  }
+}
 
+resource "aws_neptune_cluster_instance" "this" {
   auto_minor_version_upgrade   = var.auto_minor_version_upgrade
   cluster_identifier           = aws_neptune_cluster.this.cluster_identifier
+  engine                       = var.engine
   engine_version               = var.engine_version
-  instance_class               = var.enable_serverless ? "db.serverless" : var.instance_class
+  instance_class               = var.instance_class
   neptune_subnet_group_name    = aws_neptune_subnet_group.this.name
-  neptune_parameter_group_name = var.enable_serverless ? "default.neptune1.2" : module.instance_parameters[0].name
+  neptune_parameter_group_name = var.enable_serverless ? "default.neptune1.3" : aws_neptune_parameter_group.this[0].name
   port                         = var.port
+  publicly_accessible          = false
+
+  depends_on = [
+    aws_neptune_parameter_group.this
+  ]
+}
+
+resource "aws_neptune_parameter_group" "this" {
+  count = local.create_parameter_groups ? 1 : 0
+
+  name        = "${var.resource_prefix}-neptune-instance-params"
+  family      = var.parameter_group_family
+  description = "${var.resource_prefix} neptune instance-level parameter group"
+
+  parameter {
+    name  = "neptune_result_cache"
+    value = var.enable_result_cache ? "1" : "0"
+  }
 }
 
 resource "aws_kms_key" "this" {
+  count = var.create_kms_key ? 1 : 0
+
   deletion_window_in_days = 7
   description             = "Enforces encryption at rest for the ${terraform.workspace}-tier neptune cluster"
   key_usage               = "ENCRYPT_DECRYPT"
 }
 
 resource "aws_kms_alias" "this" {
+  count = var.create_kms_key ? 1 : 0
+
   name          = "alias/${var.resource_prefix}-neptune-key"
-  target_key_id = aws_kms_key.this.id
+  target_key_id = aws_kms_key.this[0].id
+}
+
+resource "aws_kms_key_policy" "this" {
+  count = var.create_kms_key ? 1 : 0
+
+  key_id = aws_kms_key.this[0].id
+  policy = data.aws_iam_policy_document.kms[0].json
 }
 
 resource "aws_neptune_subnet_group" "this" {
   name        = "${var.resource_prefix}-neptune-subnets"
   description = "subnet group for the ${terraform.workspace}-tier neptune cluster"
   subnet_ids  = var.database_subnet_ids
 }
-
-resource "aws_kms_key_policy" "this" {
-  key_id = aws_kms_key.this.id
-  policy = data.aws_iam_policy_document.kms.json
-}

terraform/modules/neptune-cluster/outputs.tf

@@ -47,73 +47,73 @@ output "cluster_reader_endpoint" {
 }
 
 output "kms_key_arn" {
-  value       = aws_kms_key.this.arn
+  value       = aws_neptune_cluster_instance.this.kms_key_arn
   description = "the neptune cluster kms key arn"
   sensitive   = false
 }
 
+output "kms_key_id" {
+  value       = local.kms_key_id
+  description = "the neptune cluster kms key id"
+  sensitive   = false
+}
+
+output "kms_alias_arn" {
+  value       = local.kms_alias_arn
+  description = "the neptune cluster kms key alias arn"
+  sensitive   = false
+}
+
+output "kms_alias_id" {
+  value       = local.kms_alias_id
+  description = "the neptune cluster kms key alias id"
+  sensitive   = false
+}
+
+output "kms_alias_name" {
+  value       = local.kms_alias_name
+  description = "the neptune cluster kms key alias name"
+  sensitive   = false
+}
+
 output "instance_address" {
-  value       = module.neptune_instance.address
+  value       = aws_neptune_cluster_instance.this.address
   description = "The hostname of the instance. See also endpoint and port."
   sensitive   = false
 }
 
 output "instance_arn" {
-  value       = module.neptune_instance.arn
+  value       = aws_neptune_cluster_instance.this.arn
   description = "The ARN of the neptune instance"
   sensitive   = false
 }
 
 output "instance_cluster_identifier" {
-  value       = module.neptune_instance.cluster_identifier
+  value       = aws_neptune_cluster_instance.this.cluster_identifier
   description = "The neptune cluster identifier"
   sensitive   = false
 }
 
 output "instance_dbi_resource_id" {
-  value       = module.neptune_instance.dbi_resource_id
+  value       = aws_neptune_cluster_instance.this.dbi_resource_id
   description = "The neptune instance resource ID"
   sensitive   = false
 }
 
 output "instance_endpoint" {
-  value       = module.neptune_instance.endpoint
+  value       = aws_neptune_cluster_instance.this.endpoint
   description = "The hostname of the instance. See also address and port."
   sensitive   = false
 }
 
 output "instance_id" {
-  value       = module.neptune_instance.id
+  value       = aws_neptune_cluster_instance.this.id
   description = "The neptune instance ID"
   sensitive   = false
 }
 
 output "instance_identifier" {
-  value       = module.neptune_instance.identifier
+  value       = aws_neptune_cluster_instance.this.identifier
   description = "The neptune instance identifier"
   sensitive   = false
 }
-
-output "kms_key_id" {
-  value       = aws_kms_key.this.key_id
-  description = "the neptune cluster kms key id"
-  sensitive   = false
-}
-
-output "kms_alias_arn" {
-  value       = aws_kms_alias.this.arn
-  description = "the neptune cluster kms key alias arn"
-  sensitive   = false
-}
-
-output "kms_alias_id" {
-  value       = aws_kms_alias.this.id
-  description = "the neptune cluster kms key alias id"
-  sensitive   = false
-}
-
-output "kms_alias_name" {
-  value       = aws_kms_alias.this.name
-  description = "the neptune cluster kms key alias name"
-  sensitive   = false
-}