feat!: Enable in org-ecs, cloudtrail-s3-sns-sqs ingestor (#103)

README.md

@@ -142,7 +142,7 @@ Some components may vary, or may be deployed on different accounts (depending on
 This would be an overall schema of the **created resources**, for the default setup.
 
 - Cloudtrail / SNS / S3 / SQS
-- SSM Parameter for Sysdig API Token Storage 
+- SSM Parameter for Sysdig API Token Storage
 - Sysdig Workload: ECS / AppRunner creation (EKS is pre-required, not created)
   - each compute solution require a role to assume for execution
 - CodeBuild for on-demand image scanning
@@ -223,6 +223,10 @@ It may take some time, but you should see logs detecting the new image in the EC
 
 ## Troubleshooting
 
+## Q-Networking: What's the requirements for the inbound/outbound connection?
+A: Refer to [Sysdig SASS Region and IP Ranges Documentation](https://docs.sysdig.com/en/docs/administration/saas-regions-and-ip-ranges/) to get Sysdig SaaS endpoint and allow both outbound (for compute vulnerability report) and inbound (for scheduled compliance checkups)
+<br/>ECS type deployment will create following [security-group setup](https://github.com/sysdiglabs/terraform-aws-secure-for-cloud/blob/master/modules/services/cloud-connector-ecs/sec-group.tf)
+
 ## Q-General: Need to modify cloud-connector config (to troubleshoot with `debug` loglevel, modify ingestors for testing, ...)
 A: both in ECS and AppRunner workload types, cloud-connector configuration is passed as a base64-encoded string through the env var `CONFIG`
 <br/>S: Get current value, decode it, edit the desired (ex.:`logging: debug` value), encode it again, and spin it again with this new definition.
@@ -245,7 +249,7 @@ A: Need to check several steps
 
 ### Q-AWS: In the ECS compute flavor of secure for cloud, I don't see any logs in the cloud-connector component
 A: This may be due to the task not beinb able to start, normally due not not having enough permissions to even fetch the secure apiToken, stored in the AWS SSM service.
-<br/>S: Access the task and see if there is any value in the "Stoped Reason" field. 
+<br/>S: Access the task and see if there is any value in the "Stopped Reason" field.
 
 ### Q-AWS: Getting error "Error: failed creating ECS Task Definition: ClientException: No Fargate configuration exists for given values.
 A: Your ECS task_size values aren't valid for Fargate. Specifically, your mem_limit value is too big for the cpu_limit you specified

examples/organizational/README.md

@@ -179,10 +179,8 @@ $ terraform apply
 |------|-------------|------|---------|:--------:|
 | <a name="input_sysdig_secure_for_cloud_member_account_id"></a> [sysdig\_secure\_for\_cloud\_member\_account\_id](#input\_sysdig\_secure\_for\_cloud\_member\_account\_id) | organizational member account where the secure-for-cloud workload is going to be deployed | `string` | n/a | yes |
 | <a name="input_benchmark_regions"></a> [benchmark\_regions](#input\_benchmark\_regions) | List of regions in which to run the benchmark. If empty, the task will contain all aws regions by default. | `list(string)` | `[]` | no |
-| <a name="input_cloudtrail_is_multi_region_trail"></a> [cloudtrail\_is\_multi\_region\_trail](#input\_cloudtrail\_is\_multi\_region\_trail) | true/false whether cloudtrail will ingest multiregional events. testing/economization purpose. | `bool` | `true` | no |
-| <a name="input_cloudtrail_kms_enable"></a> [cloudtrail\_kms\_enable](#input\_cloudtrail\_kms\_enable) | true/false whether cloudtrail delivered events to S3 should persist encrypted | `bool` | `true` | no |
-| <a name="input_cloudtrail_s3_arn"></a> [cloudtrail\_s3\_arn](#input\_cloudtrail\_s3\_arn) | ARN of a pre-existing cloudtrail\_sns s3 bucket. Used together with `cloudtrail_sns_arn`, `cloudtrail_s3_arn`. If it does not exist, it will be inferred from create cloudtrail | `string` | `"create"` | no |
-| <a name="input_cloudtrail_sns_arn"></a> [cloudtrail\_sns\_arn](#input\_cloudtrail\_sns\_arn) | ARN of a pre-existing cloudtrail\_sns. Used together with `cloudtrail_sns_arn`, `cloudtrail_s3_arn`. If it does not exist, it will be inferred from created cloudtrail. Providing an ARN requires permission to SNS:Subscribe, check ./modules/infrastructure/cloudtrail/sns\_permissions.tf block | `string` | `"create"` | no |
+| <a name="input_cloudtrail_is_multi_region_trail"></a> [cloudtrail\_is\_multi\_region\_trail](#input\_cloudtrail\_is\_multi\_region\_trail) | true/false whether the created cloudtrail will ingest multi-regional events. testing/economization purpose. | `bool` | `true` | no |
+| <a name="input_cloudtrail_kms_enable"></a> [cloudtrail\_kms\_enable](#input\_cloudtrail\_kms\_enable) | true/false whether the created cloudtrail should deliver encrypted events to s3 | `bool` | `true` | no |
 | <a name="input_connector_ecs_task_role_name"></a> [connector\_ecs\_task\_role\_name](#input\_connector\_ecs\_task\_role\_name) | Name for the ecs task role. This is only required to resolve cyclic dependency with organizational approach | `string` | `"organizational-ECSTaskRole"` | no |
 | <a name="input_deploy_benchmark"></a> [deploy\_benchmark](#input\_deploy\_benchmark) | Whether to deploy or not the cloud benchmarking | `bool` | `true` | no |
 | <a name="input_deploy_image_scanning_ecr"></a> [deploy\_image\_scanning\_ecr](#input\_deploy\_image\_scanning\_ecr) | true/false whether to deploy the image scanning on ECR pushed images | `bool` | `false` | no |
@@ -193,6 +191,7 @@ $ terraform apply
 | <a name="input_ecs_vpc_id"></a> [ecs\_vpc\_id](#input\_ecs\_vpc\_id) | ID of the VPC where the workload is to be deployed. Defaulted to be created when `ecs_cluster_name is not provided.` | `string` | `"create"` | no |
 | <a name="input_ecs_vpc_region_azs"></a> [ecs\_vpc\_region\_azs](#input\_ecs\_vpc\_region\_azs) | List of Availability Zones for ECS VPC creation. e.g.: ["apne1-az1", "apne1-az2"]. If defaulted, two of the default 'aws\_availability\_zones' datasource will be taken | `list(string)` | `[]` | no |
 | <a name="input_ecs_vpc_subnets_private_ids"></a> [ecs\_vpc\_subnets\_private\_ids](#input\_ecs\_vpc\_subnets\_private\_ids) | List of VPC subnets where workload is to be deployed. Defaulted to be created when `ecs_cluster_name is not provided.` | `list(string)` | `[]` | no |
+| <a name="input_existing_cloudtrail_config"></a> [existing\_cloudtrail\_config](#input\_existing\_cloudtrail\_config) | Optional block. If not set, a new cloudtrail, sns and sqs resources will be created<br/><br>If there's an existing cloudtrail, input mandatory attributes, and one of the 1, 2 or 3 grouped labeled optionals.<br><ul><br>  <li>cloudtrail\_s3\_arn: Mandatory ARN of a pre-existing cloudtrail\_sns s3 bucket. Used together with `cloudtrail_sns_arn`, `cloudtrail_s3_arn`. If it does not exist, it will be inferred from create cloudtrail"</li><br>  <li>cloudtrail\_sns\_arn: Optional 1. ARN of a pre-existing cloudtrail\_sns. Used together with `cloudtrail_sns_arn`, `cloudtrail_s3_arn`. If it does not exist, it will be inferred from created cloudtrail. Providing an ARN requires permission to SNS:Subscribe, check ./modules/infrastructure/cloudtrail/sns\_permissions.tf block</li><br>  <li>cloudtrail\_s3\_role\_arn: Optional 2. ARN of the role to be assumed for S3 access. This role must be in the same account of the S3 bucket. Currently this setup is not compatible with organizational scanning feature</li><br>  <li>cloudtrail\_s3\_sns\_sqs\_arn: Optional 3. ARN of the queue that will ingest events forwarded from an existing cloudtrail\_s3\_sns</li><br>  <li>cloudtrail\_s3\_sns\_sqs\_url: Optional 3. URL of the queue that will ingest events forwarded from an existing cloudtrail\_s3\_sns<</li><br></ul> | <pre>object({<br>    cloudtrail_s3_arn         = optional(string)<br>    cloudtrail_sns_arn        = optional(string)<br>    cloudtrail_s3_role_arn    = optional(string)<br>    cloudtrail_s3_sns_sqs_arn = optional(string)<br>    cloudtrail_s3_sns_sqs_url = optional(string)<br>  })</pre> | <pre>{<br>  "cloudtrail_s3_arn": "create",<br>  "cloudtrail_s3_role_arn": null,<br>  "cloudtrail_s3_sns_sqs_arn": null,<br>  "cloudtrail_s3_sns_sqs_url": null,<br>  "cloudtrail_sns_arn": "create"<br>}</pre> | no |
 | <a name="input_name"></a> [name](#input\_name) | Name to be assigned to all child resources. A suffix may be added internally when required. Use default value unless you need to install multiple instances | `string` | `"sfc"` | no |
 | <a name="input_organizational_member_default_admin_role"></a> [organizational\_member\_default\_admin\_role](#input\_organizational\_member\_default\_admin\_role) | Default role created by AWS for management-account users to be able to admin member accounts.<br/>https://docs.aws.amazon.com/organizations/latest/userguide/orgs_manage_accounts_access.html | `string` | `"OrganizationAccountAccessRole"` | no |
 | <a name="input_tags"></a> [tags](#input\_tags) | sysdig secure-for-cloud tags. always include 'product' default tag for resource-group proper functioning | `map(string)` | <pre>{<br>  "product": "sysdig-secure-for-cloud"<br>}</pre> | no |

examples/organizational/cloudtrail.tf

@@ -1,12 +1,12 @@
 locals {
-  cloudtrail_deploy  = var.cloudtrail_sns_arn == "create"
-  cloudtrail_sns_arn = local.cloudtrail_deploy ? module.cloudtrail[0].sns_topic_arn : var.cloudtrail_sns_arn
-  cloudtrail_s3_arn  = local.cloudtrail_deploy ? module.cloudtrail[0].s3_bucket_arn : var.cloudtrail_s3_arn
+  deploy_cloudtrail  = var.existing_cloudtrail_config == null || var.existing_cloudtrail_config.cloudtrail_sns_arn == "create" || var.existing_cloudtrail_config.cloudtrail_sns_arn == null
+  cloudtrail_sns_arn = local.deploy_cloudtrail ? module.cloudtrail[0].cloudtrail_sns_arn : var.existing_cloudtrail_config.cloudtrail_sns_arn
+  cloudtrail_s3_arn  = local.deploy_cloudtrail ? module.cloudtrail[0].s3_bucket_arn : var.existing_cloudtrail_config.cloudtrail_s3_arn
 }
 
 
 module "cloudtrail" {
-  count  = local.cloudtrail_deploy ? 1 : 0
+  count  = local.deploy_cloudtrail ? 1 : 0
   source = "../../modules/infrastructure/cloudtrail"
   name   = var.name
 

examples/organizational/main.tf

@@ -69,15 +69,20 @@ module "cloud_connector" {
 
   is_organizational = true
   organizational_config = {
-    sysdig_secure_for_cloud_role_arn = module.secure_for_cloud_role.sysdig_secure_for_cloud_role_arn
+    # see local.deploy_org_management_sysdig_role notes
+    sysdig_secure_for_cloud_role_arn = local.deploy_org_management_sysdig_role ? module.secure_for_cloud_role[0].sysdig_secure_for_cloud_role_arn : var.existing_cloudtrail_config.cloudtrail_s3_role_arn
     organizational_role_per_account  = var.organizational_member_default_admin_role
     connector_ecs_task_role_name     = aws_iam_role.connector_ecs_task.name
   }
 
   build_project_arn  = length(module.codebuild) == 1 ? module.codebuild[0].project_arn : "na"
   build_project_name = length(module.codebuild) == 1 ? module.codebuild[0].project_name : "na"
 
-  sns_topic_arn = local.cloudtrail_sns_arn
+  existing_cloudtrail_config = {
+    cloudtrail_sns_arn        = local.cloudtrail_sns_arn
+    cloudtrail_s3_sns_sqs_url = var.existing_cloudtrail_config.cloudtrail_s3_sns_sqs_url
+    cloudtrail_s3_sns_sqs_arn = var.existing_cloudtrail_config.cloudtrail_s3_sns_sqs_arn
+  }
 
   ecs_cluster_name            = local.ecs_cluster_name
   ecs_vpc_id                  = local.ecs_vpc_id

examples/organizational/permissions.tf

@@ -1,4 +1,11 @@
+locals {
+  # only deploy org-management-account lvl role if scanning is deployed and we're not overriding S3Role
+  # FIXME. main.tf#72  if scanning is activated, using 'cloudtrail_s3_role_arn' won't work, FR: need to provision 2 roles in cloud-connector
+  deploy_org_management_sysdig_role = var.deploy_image_scanning_ecs || var.deploy_image_scanning_ecr || var.existing_cloudtrail_config.cloudtrail_s3_sns_sqs_arn == null
+}
+
 module "secure_for_cloud_role" {
+  count  = local.deploy_org_management_sysdig_role ? 1 : 0
   source = "../../modules/infrastructure/permissions/org-role-ecs"
   providers = {
     aws.member = aws.member
@@ -18,7 +25,7 @@ module "secure_for_cloud_role" {
 # secure_for_cloud_role <-> ecs_role trust relationship
 # note:
 # - definition of a ROOT lvl secure_for_cloud_connector_ecs_tas_role to avoid cyclic dependencies
-# - duplicated in ../../modules/services/cloud-connector-ecs/ecs-service-security.tf
+# - duplicated in ../../modules/services/cloud-connector-ecs/permissions.tf
 # -----------------------------------------------------------------
 resource "aws_iam_role" "connector_ecs_task" {
   provider           = aws.member

examples/organizational/variables.tf

@@ -28,31 +28,49 @@ variable "organizational_member_default_admin_role" {
 #
 # cloudtrail configuration
 #
-
-variable "cloudtrail_sns_arn" {
-  type        = string
-  default     = "create"
-  description = "ARN of a pre-existing cloudtrail_sns. Used together with `cloudtrail_sns_arn`, `cloudtrail_s3_arn`. If it does not exist, it will be inferred from created cloudtrail. Providing an ARN requires permission to SNS:Subscribe, check ./modules/infrastructure/cloudtrail/sns_permissions.tf block"
-}
-
-variable "cloudtrail_s3_arn" {
-  type        = string
-  default     = "create"
-  description = "ARN of a pre-existing cloudtrail_sns s3 bucket. Used together with `cloudtrail_sns_arn`, `cloudtrail_s3_arn`. If it does not exist, it will be inferred from create cloudtrail"
-}
-
 variable "cloudtrail_is_multi_region_trail" {
   type        = bool
   default     = true
-  description = "true/false whether cloudtrail will ingest multiregional events. testing/economization purpose."
+  description = "true/false whether the created cloudtrail will ingest multi-regional events. testing/economization purpose."
 }
 
 variable "cloudtrail_kms_enable" {
   type        = bool
   default     = true
-  description = "true/false whether cloudtrail delivered events to S3 should persist encrypted"
+  description = "true/false whether the created cloudtrail should deliver encrypted events to s3"
+}
+
+
+variable "existing_cloudtrail_config" {
+  type = object({
+    cloudtrail_s3_arn         = optional(string)
+    cloudtrail_sns_arn        = optional(string)
+    cloudtrail_s3_role_arn    = optional(string)
+    cloudtrail_s3_sns_sqs_arn = optional(string)
+    cloudtrail_s3_sns_sqs_url = optional(string)
+  })
+  default = {
+    cloudtrail_s3_arn         = "create"
+    cloudtrail_sns_arn        = "create"
+    cloudtrail_s3_role_arn    = null
+    cloudtrail_s3_sns_sqs_arn = null
+    cloudtrail_s3_sns_sqs_url = null
+  }
+
+  description = <<-EOT
+    Optional block. If not set, a new cloudtrail, sns and sqs resources will be created<br/>
+    If there's an existing cloudtrail, input mandatory attributes, and one of the 1, 2 or 3 grouped labeled optionals.
+    <ul>
+      <li>cloudtrail_s3_arn: Mandatory ARN of a pre-existing cloudtrail_sns s3 bucket. Used together with `cloudtrail_sns_arn`, `cloudtrail_s3_arn`. If it does not exist, it will be inferred from create cloudtrail"</li>
+      <li>cloudtrail_sns_arn: Optional 1. ARN of a pre-existing cloudtrail_sns. Used together with `cloudtrail_sns_arn`, `cloudtrail_s3_arn`. If it does not exist, it will be inferred from created cloudtrail. Providing an ARN requires permission to SNS:Subscribe, check ./modules/infrastructure/cloudtrail/sns_permissions.tf block</li>
+      <li>cloudtrail_s3_role_arn: Optional 2. ARN of the role to be assumed for S3 access. This role must be in the same account of the S3 bucket. Currently this setup is not compatible with organizational scanning feature</li>
+      <li>cloudtrail_s3_sns_sqs_arn: Optional 3. ARN of the queue that will ingest events forwarded from an existing cloudtrail_s3_sns</li>
+      <li>cloudtrail_s3_sns_sqs_url: Optional 3. URL of the queue that will ingest events forwarded from an existing cloudtrail_s3_sns<</li>
+    </ul>
+  EOT
 }
 
+
 #
 # scanning configuration
 #

examples/organizational/versions.tf

@@ -1,5 +1,6 @@
 terraform {
   required_version = ">= 1.0.0"
+  experiments      = [module_variable_optional_attrs]
   required_providers {
     aws = {
       version               = ">= 4.0.0"

examples/single-account-apprunner/cloudtrail.tf

@@ -1,6 +1,6 @@
 locals {
   cloudtrail_deploy  = var.cloudtrail_sns_arn == "create"
-  cloudtrail_sns_arn = local.cloudtrail_deploy ? module.cloudtrail[0].sns_topic_arn : var.cloudtrail_sns_arn
+  cloudtrail_sns_arn = local.cloudtrail_deploy ? module.cloudtrail[0].cloudtrail_sns_arn : var.cloudtrail_sns_arn
 }
 
 module "cloudtrail" {

examples/single-account-apprunner/outputs.tf

@@ -1,4 +1,4 @@
 output "cloudtrail_sns_topic_arn" {
-  value       = length(module.cloudtrail) > 0 ? module.cloudtrail[0].sns_topic_arn : var.cloudtrail_sns_arn
+  value       = length(module.cloudtrail) > 0 ? module.cloudtrail[0].cloudtrail_sns_arn : var.cloudtrail_sns_arn
   description = "ARN of cloudtrail_sns topic"
 }

examples/single-account-ecs/cloudtrail.tf

@@ -1,6 +1,6 @@
 locals {
   cloudtrail_deploy  = var.cloudtrail_sns_arn == "create"
-  cloudtrail_sns_arn = local.cloudtrail_deploy ? module.cloudtrail[0].sns_topic_arn : var.cloudtrail_sns_arn
+  cloudtrail_sns_arn = local.cloudtrail_deploy ? module.cloudtrail[0].cloudtrail_sns_arn : var.cloudtrail_sns_arn
 }
 
 module "cloudtrail" {

examples/single-account-ecs/main.tf

@@ -51,7 +51,10 @@ module "cloud_connector" {
   build_project_arn  = length(module.codebuild) == 1 ? module.codebuild[0].project_arn : "na"
   build_project_name = length(module.codebuild) == 1 ? module.codebuild[0].project_name : "na"
 
-  sns_topic_arn = local.cloudtrail_sns_arn
+  existing_cloudtrail_config = {
+    cloudtrail_sns_arn = local.cloudtrail_sns_arn
+  }
+
 
   ecs_cluster_name            = local.ecs_cluster_name
   ecs_vpc_id                  = local.ecs_vpc_id

examples/single-account-ecs/outputs.tf

@@ -1,4 +1,4 @@
 output "cloudtrail_sns_topic_arn" {
-  value       = length(module.cloudtrail) > 0 ? module.cloudtrail[0].sns_topic_arn : var.cloudtrail_sns_arn
+  value       = length(module.cloudtrail) > 0 ? module.cloudtrail[0].cloudtrail_sns_arn : var.cloudtrail_sns_arn
   description = "ARN of cloudtrail_sns topic"
 }

examples/single-account-k8s/cloud-connector.tf

@@ -8,9 +8,9 @@ locals {
 module "cloud_connector_sqs" {
   source = "../../modules/infrastructure/sqs-sns-subscription"
 
-  name          = var.name
-  sns_topic_arn = local.cloudtrail_sns_arn
-  tags          = var.tags
+  name               = var.name
+  cloudtrail_sns_arn = local.cloudtrail_sns_arn
+  tags               = var.tags
 }
 
 module "codebuild" {

examples/single-account-k8s/cloudtrail.tf

@@ -1,6 +1,6 @@
 locals {
   cloudtrail_deploy  = var.cloudtrail_sns_arn == "create"
-  cloudtrail_sns_arn = var.cloudtrail_sns_arn == "create" ? module.cloudtrail[0].sns_topic_arn : var.cloudtrail_sns_arn
+  cloudtrail_sns_arn = var.cloudtrail_sns_arn == "create" ? module.cloudtrail[0].cloudtrail_sns_arn : var.cloudtrail_sns_arn
 }
 
 module "cloudtrail" {

modules/infrastructure/cloudtrail/README.md

@@ -53,8 +53,8 @@ No modules.
 
 | Name | Description |
 |------|-------------|
+| <a name="output_cloudtrail_sns_arn"></a> [cloudtrail\_sns\_arn](#output\_cloudtrail\_sns\_arn) | ARN of Cloudtrail SNS topic |
 | <a name="output_s3_bucket_arn"></a> [s3\_bucket\_arn](#output\_s3\_bucket\_arn) | ARN of Cloudtrail SNS topic |
-| <a name="output_sns_topic_arn"></a> [sns\_topic\_arn](#output\_sns\_topic\_arn) | ARN of Cloudtrail SNS topic |
 <!-- END OF PRE-COMMIT-TERRAFORM DOCS HOOK -->
 
 ## Authors

modules/infrastructure/cloudtrail/outputs.tf

@@ -1,4 +1,4 @@
-output "sns_topic_arn" {
+output "cloudtrail_sns_arn" {
   value       = aws_sns_topic.cloudtrail.arn
   description = "ARN of Cloudtrail SNS topic"
 }

modules/infrastructure/cloudtrail_s3-sns-sqs/main.tf

@@ -47,9 +47,9 @@ resource "aws_s3_bucket_notification" "bucket_notification" {
 # sqs
 # --------------------
 module "cloudtrail_s3_sns_sqs" {
-  source        = "../sqs-sns-subscription"
-  name          = "${var.name}-s3-sqs"
-  sns_topic_arn = aws_sns_topic.s3_sns.arn
+  source             = "../sqs-sns-subscription"
+  name               = "${var.name}-s3-sqs"
+  cloudtrail_sns_arn = aws_sns_topic.s3_sns.arn
 
   tags = var.tags
 }

modules/infrastructure/permissions/org-role-ecs/README.md

@@ -7,6 +7,7 @@ The aim of this module is to manage the organizational **managed account** requi
 
 
 ## Permissions
+
 * Threat-Detection feature
   * S3 Get and List permissions in order to fetch the events
   * SNS Subscription permissions in order to subscribe a topic to it