diff --git a/README.md b/README.md index 8ca753f..a53f4b2 100644 --- a/README.md +++ b/README.md @@ -3,12 +3,12 @@ [![Build Status](https://github.com/mineiros-io/terraform-google-logging-sink/workflows/Tests/badge.svg)](https://github.com/mineiros-io/terraform-google-logging-sink/actions) [![GitHub tag (latest SemVer)](https://img.shields.io/github/v/tag/mineiros-io/terraform-google-logging-sink.svg?label=latest&sort=semver)](https://github.com/mineiros-io/terraform-google-logging-sink/releases) [![Terraform Version](https://img.shields.io/badge/Terraform-1.x-623CE4.svg?logo=terraform)](https://github.com/hashicorp/terraform/releases) -[![AWS Provider Version](https://img.shields.io/badge/AWS-3-F8991D.svg?logo=terraform)](https://github.com/terraform-providers/terraform-provider-aws/releases) +[![Google Provider Version](https://img.shields.io/badge/google-4-1A73E8.svg?logo=terraform)](https://github.com/terraform-providers/terraform-provider-google/releases) [![Join Slack](https://img.shields.io/badge/slack-@mineiros--community-f32752.svg?logo=slack)](https://mineiros.io/slack) # terraform-google-logging-sink -A [Terraform] module for [Amazon Web Services (AWS)][aws]. +A [Terraform](https://www.terraform.io) module to create and manage [Google Project Logging Sinks](https://cloud.google.com/logging/docs/reference/v2/rest/v2/projects.sinks). **_This module supports Terraform version 1 and is compatible with the Terraform AWS Provider version 3._** @@ -39,11 +39,7 @@ secure, and production-grade cloud infrastructure. This module implements the following Terraform resources: -- `null_resource` - -and supports additional features of the following modules: - -- [mineiros-io/something/google](https://github.com/mineiros-io/terraform-google-something) +- `google_logging_project_sink` ## Getting Started @@ -52,6 +48,9 @@ Most common usage of the module: ```hcl module "terraform-google-logging-sink" { source = "git@github.com:mineiros-io/terraform-google-logging-sink.git?ref=v0.0.1" + + name = "my-pubsub-instance-sink" + destination = "pubsub.googleapis.com/projects/my-project/topics/instance-activity" } ``` @@ -61,40 +60,93 @@ See [variables.tf] and [examples/] for details and use-cases. ### Main Resource Configuration -- [**`example_required`**](#var-example_required): *(**Required** `string`)* +- [**`name`**](#var-name): *(**Required** `string`)* - The name of the resource + The name of the logging sink. -- [**`example_name`**](#var-example_name): *(Optional `string`)* +- [**`destination`**](#var-destination): *(**Required** `string`)* - The name of the resource + The destination of the sink (or, in other words, where logs are written to). - Default is `"optional-resource-name"`. + Can be a Cloud Storage bucket, a PubSub topic, a BigQuery dataset or a Cloud Logging bucket. -- [**`example_user_object`**](#var-example_user_object): *(Optional `object(user)`)* + Examples: + - `"storage.googleapis.com/[GCS_BUCKET]"` + - `"bigquery.googleapis.com/projects/[PROJECT_ID]/datasets/[DATASET]"` + - `"pubsub.googleapis.com/projects/[PROJECT_ID]/topics/[TOPIC_ID]"` + - `"logging.googleapis.com/projects/[PROJECT_ID]]/locations/global/buckets/[BUCKET_ID]"` - Default is `{}`. + The writer associated with the sink must have access to write to the above resource. - Example: +- [**`filter`**](#var-filter): *(Optional `string`)* - ```hcl - user = { - name = "marius" - description = "The guy from Berlin." - } - ``` + The filter to apply when exporting logs. Only log entries that match the filter are exported. + + See [Advanced Log Filters](https://cloud.google.com/logging/docs/view/building-queries) for information on how to write a filter. + +- [**`description`**](#var-description): *(Optional `string`)* + + A description of this sink. The maximum length of the description is 8000 characters. + +- [**`disabled`**](#var-disabled): *(Optional `bool`)* + + If set to True, then this sink is disabled and it does not export any log entries. + +- [**`project`**](#var-project): *(Optional `string`)* + + The ID of the project to create the sink in. + + If omitted, the project associated with the provider is used. + +- [**`unique_writer_identity`**](#var-unique_writer_identity): *(Optional `bool`)* + + Whether or not to create a unique identity associated with this sink. + + If `false` (the default), then the `writer_identity` used is `serviceAccount:cloud-logs@system.gserviceaccount.com`. + + If `true`, then a unique service account is created and used for this sink. If you wish to publish logs across projects or utilize `bigquery_options`, you must set `unique_writer_identity` to true. + + Default is `false`. + +- [**`use_partitioned_tables`**](#var-use_partitioned_tables): *(Optional `bool`)* + + Whether to use [BigQuery's partition tables](https://cloud.google.com/bigquery/docs/partitioned-tables). + + By default, Logging creates dated tables based on the log entries' timestamps, e.g. syslog_20170523. With partitioned tables the date suffix is no longer present and [special query syntax](https://cloud.google.com/bigquery/docs/querying-partitioned-tables) has to be used instead. In both cases, tables are sharded based on UTC timezone. + + Default is `null`. + +- [**`exclusions`**](#var-exclusions): *(Optional `list(exclusion)`)* + + Log entries that match any of the exclusion filters will not be exported. + + If a log entry is matched by both filter and one of `exclusion_filters` it will not be exported. Can be repeated multiple times for multiple exclusions. + + Default is `[]`. + + Each `exclusion` object in the list accepts the following attributes: + + - [**`name`**](#attr-exclusions-name): *(**Required** `string`)* + + A client-assigned identifier, such as `load-balancer-exclusion`. + + Identifiers are limited to 100 characters and can include only letters, digits, underscores, hyphens, and periods. First character has to be alphanumeric. + + The object accepts the following attributes: + + - [**`description`**](#attr-exclusions-name-description): *(Optional `string`)* - The `user` object accepts the following attributes: + A description of this exclusion. - - [**`name`**](#attr-example_user_object-name): *(**Required** `string`)* + - [**`filter`**](#attr-exclusions-name-filter): *(**Required** `string`)* - The name of the user + An advanced logs filter that matches the log entries to be excluded. By using the sample function, you can exclude less than 100% of the matching log entries. - - [**`description`**](#attr-example_user_object-description): *(Optional `string`)* + See [Advanced Log Filters](https://cloud.google.com/logging/docs/view/advanced_filters) for information on how to write a filter. - A description describng the user in more detail + - [**`disabled`**](#attr-exclusions-name-disabled): *(Optional `bool`)* - Default is `""`. + If set to `true`, then this exclusion is disabled and it does not exclude any log entries. ### Module Configuration diff --git a/README.tfdoc.hcl b/README.tfdoc.hcl index 397129c..92dd562 100644 --- a/README.tfdoc.hcl +++ b/README.tfdoc.hcl @@ -20,25 +20,12 @@ header { text = "Terraform Version" } - # TODO: remove and enable gh or gcp provider badge - badge "tf-aws-provider" { - image = "https://img.shields.io/badge/AWS-3-F8991D.svg?logo=terraform" - url = "https://github.com/terraform-providers/terraform-provider-aws/releases" - text = "AWS Provider Version" + badge "tf-gcp-provider" { + image = "https://img.shields.io/badge/google-4-1A73E8.svg?logo=terraform" + url = "https://github.com/terraform-providers/terraform-provider-google/releases" + text = "Google Provider Version" } - # badge "tf-gh" { - # image = "https://img.shields.io/badge/GH-4-F8991D.svg?logo=terraform" - # url = "https://github.com/terraform-providers/terraform-provider-github/releases" - # text = "Github Provider Version" - # } - - # badge "tf-gcp-provider" { - # image = "https://img.shields.io/badge/google-4-1A73E8.svg?logo=terraform" - # url = "https://github.com/terraform-providers/terraform-provider-google/releases" - # text = "Google Provider Version" - # } - badge "slack" { image = "https://img.shields.io/badge/slack-@mineiros--community-f32752.svg?logo=slack" url = "https://mineiros.io/slack" @@ -50,7 +37,7 @@ section { title = "terraform-google-logging-sink" toc = true content = <<-END - A [Terraform] module for [Amazon Web Services (AWS)][aws]. + A [Terraform](https://www.terraform.io) module to create and manage [Google Project Logging Sinks](https://cloud.google.com/logging/docs/reference/v2/rest/v2/projects.sinks). **_This module supports Terraform version 1 and is compatible with the Terraform AWS Provider version 3._** @@ -65,11 +52,7 @@ section { content = <<-END This module implements the following Terraform resources: - - `null_resource` - - and supports additional features of the following modules: - - - [mineiros-io/something/google](https://github.com/mineiros-io/terraform-google-something) + - `google_logging_project_sink` END } @@ -81,6 +64,9 @@ section { ```hcl module "terraform-google-logging-sink" { source = "git@github.com:mineiros-io/terraform-google-logging-sink.git?ref=v0.0.1" + + name = "my-pubsub-instance-sink" + destination = "pubsub.googleapis.com/projects/my-project/topics/instance-activity" } ``` END @@ -95,53 +81,132 @@ section { section { title = "Main Resource Configuration" - # please add main resource variables here - - # TODO: remove examples + variable "name" { + required = true + type = string + description = "The name of the logging sink." + } - ### Example of a required variable - variable "example_required" { + variable "destination" { required = true type = string description = <<-END - The name of the resource + The destination of the sink (or, in other words, where logs are written to). + + Can be a Cloud Storage bucket, a PubSub topic, a BigQuery dataset or a Cloud Logging bucket. + + Examples: + - `"storage.googleapis.com/[GCS_BUCKET]"` + - `"bigquery.googleapis.com/projects/[PROJECT_ID]/datasets/[DATASET]"` + - `"pubsub.googleapis.com/projects/[PROJECT_ID]/topics/[TOPIC_ID]"` + - `"logging.googleapis.com/projects/[PROJECT_ID]]/locations/global/buckets/[BUCKET_ID]"` + + The writer associated with the sink must have access to write to the above resource. END } - ### Example of an optional variable - variable "example_name" { + variable "filter" { type = string description = <<-END - The name of the resource + The filter to apply when exporting logs. Only log entries that match the filter are exported. + + See [Advanced Log Filters](https://cloud.google.com/logging/docs/view/building-queries) for information on how to write a filter. END - default = "optional-resource-name" } - ### Example of an object - variable "example_user_object" { - type = object(user) - default = {} - readme_example = <<-END - user = { - name = "marius" - description = "The guy from Berlin." - } + variable "description" { + type = string + description = "A description of this sink. The maximum length of the description is 8000 characters." + } + + variable "disabled" { + type = bool + description = "If set to True, then this sink is disabled and it does not export any log entries." + } + + variable "project" { + type = string + description = <<-END + The ID of the project to create the sink in. + + If omitted, the project associated with the provider is used. + END + } + + variable "unique_writer_identity" { + type = bool + description = <<-END + Whether or not to create a unique identity associated with this sink. + + If `false` (the default), then the `writer_identity` used is `serviceAccount:cloud-logs@system.gserviceaccount.com`. + + If `true`, then a unique service account is created and used for this sink. If you wish to publish logs across projects or utilize `bigquery_options`, you must set `unique_writer_identity` to true. + END + default = false + } + + # TODO: remove if we decide to go with `var.use_partitioned_tables` instead + # variable "bigquery_options" { + # type = object(option) + # description = "Options that affect sinks exporting data to BigQuery." + + # attribute "use_partitioned_tables" { + # required = true + # type = bool + # description = <<-END + # Whether to use [BigQuery's partition tables](https://cloud.google.com/bigquery/docs/partitioned-tables). + + # By default, Logging creates dated tables based on the log entries' timestamps, e.g. syslog_20170523. With partitioned tables the date suffix is no longer present and [special query syntax](https://cloud.google.com/bigquery/docs/querying-partitioned-tables) has to be used instead. In both cases, tables are sharded based on UTC timezone. + # END + # } + # } + variable "use_partitioned_tables" { + type = bool + description = <<-END + Whether to use [BigQuery's partition tables](https://cloud.google.com/bigquery/docs/partitioned-tables). + + By default, Logging creates dated tables based on the log entries' timestamps, e.g. syslog_20170523. With partitioned tables the date suffix is no longer present and [special query syntax](https://cloud.google.com/bigquery/docs/querying-partitioned-tables) has to be used instead. In both cases, tables are sharded based on UTC timezone. END + default = null + } + + variable "exclusions" { + type = list(exclusion) + description = <<-END + Log entries that match any of the exclusion filters will not be exported. + + If a log entry is matched by both filter and one of `exclusion_filters` it will not be exported. Can be repeated multiple times for multiple exclusions. + END + default = [] attribute "name" { required = true type = string description = <<-END - The name of the user - END - } + A client-assigned identifier, such as `load-balancer-exclusion`. - attribute "description" { - type = string - default = "" - description = <<-END - A description describng the user in more detail + Identifiers are limited to 100 characters and can include only letters, digits, underscores, hyphens, and periods. First character has to be alphanumeric. END + + attribute "description" { + type = string + description = "A description of this exclusion." + } + + attribute "filter" { + required = true + type = string + description = <<-END + An advanced logs filter that matches the log entries to be excluded. By using the sample function, you can exclude less than 100% of the matching log entries. + + See [Advanced Log Filters](https://cloud.google.com/logging/docs/view/advanced_filters) for information on how to write a filter. + END + } + + attribute "disabled" { + type = bool + description = "If set to `true`, then this exclusion is disabled and it does not exclude any log entries." + } } } } diff --git a/main.tf b/main.tf index 6a5c773..0ecc7df 100644 --- a/main.tf +++ b/main.tf @@ -3,3 +3,37 @@ # And it continues with some lowercase information about the module # We might add more than one line for additional information # ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +resource "google_logging_project_sink" "sink" { + count = var.module_enabled ? 1 : 0 + + name = var.name + destination = var.destination + + filter = var.filter + description = var.description + disabled = var.disabled + project = var.project + unique_writer_identity = var.unique_writer_identity + + dynamic "bigquery_options" { + for_each = var.use_partitioned_tables != null ? [1] : [] + + content { + use_partitioned_tables = var.use_partitioned_tables + } + } + + dynamic "exclusions" { + for_each = var.exclusions + iterator = exclusion + + content { + name = exclusion.value.name + filter = exclusion.value.filter + description = try(exclusion.value.description, null) + } + } + + depends_on = [var.module_depends_on] +} diff --git a/outputs.tf b/outputs.tf index 42c877c..940de94 100644 --- a/outputs.tf +++ b/outputs.tf @@ -2,6 +2,12 @@ # OUTPUT CALCULATED VARIABLES (prefer full objects) # ---------------------------------------------------------------------------------------------------------------------- +output "sink" { + description = "All attributes of the created `google_logging_project_sink` resource." + value = try(google_logging_project_sink.sink[0], null) +} + + # ---------------------------------------------------------------------------------------------------------------------- # OUTPUT ALL RESOURCES AS FULL OBJECTS # ---------------------------------------------------------------------------------------------------------------------- @@ -14,8 +20,3 @@ output "module_enabled" { description = "Whether or not the module is enabled." value = var.module_enabled } - -# output "module_tags" { -# description = "A map of tags that will be applied to all created resources that accept tags." -# value = var.module_tags -# } diff --git a/test/unit-complete/main.tf b/test/unit-complete/main.tf index 04ea55b..6cd54d6 100644 --- a/test/unit-complete/main.tf +++ b/test/unit-complete/main.tf @@ -4,25 +4,15 @@ # The purpose is to activate everything the module offers, but trying to keep execution time and costs minimal. # ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -variable "aws_region" { - description = "(Optional) The AWS region in which all resources will be created." - type = string - default = "us-east-1" -} - terraform { required_providers { - aws = { - source = "hashicorp/aws" - version = "~> 3.0" + google = { + source = "hashicorp/google" + version = "4.14.0" } } } -provider "aws" { - region = var.aws_region -} - # DO NOT RENAME MODULE NAME module "test" { source = "../.." @@ -30,14 +20,24 @@ module "test" { module_enabled = true # add all required arguments + name = "my-pubsub-instance-sink" + destination = "pubsub.googleapis.com/projects/my-project/topics/instance-activity" # add all optional arguments that create additional resources - - # add most/all other optional arguments - - # module_tags = { - # Environment = "unknown" - # } + description = "some explanation on what this is" + filter = "resource.type = gce_instance" + unique_writer_identity = true + use_partitioned_tables = true + exclusions = [{ + name = "nsexcllusion1" + description = "Exclude logs from namespace-1 in k8s" + filter = "resource.type = k8s_container resource.labels.namespace_name=\"namespace-1\" " + }, + { + name = "nsexcllusion2" + description = "Exclude logs from namespace-2 in k8s" + filter = "resource.type = k8s_container resource.labels.namespace_name=\"namespace-2\" " + }] module_depends_on = ["nothing"] } diff --git a/test/unit-disabled/main.tf b/test/unit-disabled/main.tf index 5b0ee91..66b946d 100644 --- a/test/unit-disabled/main.tf +++ b/test/unit-disabled/main.tf @@ -4,25 +4,15 @@ # The purpose is to verify no resources are created when the module is disabled. # ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -variable "aws_region" { - description = "(Optional) The AWS region in which all resources will be created." - type = string - default = "us-east-1" -} - terraform { required_providers { - aws = { - source = "hashicorp/aws" - version = "~> 3.0" + google = { + source = "hashicorp/google" + version = "4.14.0" } } } -provider "aws" { - region = var.aws_region -} - # DO NOT RENAME MODULE NAME module "test" { source = "../.." @@ -30,6 +20,8 @@ module "test" { module_enabled = false # add all required arguments + name = "my-pubsub-instance-sink" + destination = "pubsub.googleapis.com/projects/my-project/topics/instance-activity" # add all optional arguments that create additional resources } diff --git a/test/unit-minimal/main.tf b/test/unit-minimal/main.tf index 8917e41..7939714 100644 --- a/test/unit-minimal/main.tf +++ b/test/unit-minimal/main.tf @@ -4,32 +4,21 @@ # The purpose is to test all defaults for optional arguments and just provide the required arguments. # ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -variable "aws_region" { - description = "(Optional) The AWS region in which all resources will be created." - type = string - default = "us-east-1" -} - terraform { required_providers { - aws = { - source = "hashicorp/aws" - # always test with exact version to catch unsupported blocks/arguments early - # this should match the minimal version in versions.tf - version = "3.0.0" + google = { + source = "hashicorp/google" + version = "4.14.0" } } } -provider "aws" { - region = var.aws_region -} - # DO NOT RENAME MODULE NAME module "test" { source = "../.." - # add only required arguments and no optional arguments + name = "my-pubsub-instance-sink" + destination = "pubsub.googleapis.com/projects/my-project/topics/instance-activity" } # outputs generate non-idempotent terraform plans so we disable them for now unless we need them. diff --git a/variables.tf b/variables.tf index 433f452..dda067e 100644 --- a/variables.tf +++ b/variables.tf @@ -3,11 +3,105 @@ # These variables must be set when using this module. # ---------------------------------------------------------------------------------------------------------------------- +variable "name" { + type = string + description = "(Required) The name of the logging sink." +} + +variable "destination" { + type = string + description = <<-END + (Required) The destination of the sink (or, in other words, where logs are written to). + + Can be a Cloud Storage bucket, a PubSub topic, a BigQuery dataset or a Cloud Logging bucket. + + Examples: + - `"storage.googleapis.com/[GCS_BUCKET]"` + - `"bigquery.googleapis.com/projects/[PROJECT_ID]/datasets/[DATASET]"` + - `"pubsub.googleapis.com/projects/[PROJECT_ID]/topics/[TOPIC_ID]"` + - `"logging.googleapis.com/projects/[PROJECT_ID]]/locations/global/buckets/[BUCKET_ID]"` + + The writer associated with the sink must have access to write to the above resource. + END +} + # ---------------------------------------------------------------------------------------------------------------------- # OPTIONAL PARAMETERS # These variables have defaults, but may be overridden. # ---------------------------------------------------------------------------------------------------------------------- +variable "filter" { + type = string + description = "(Optional) The filter to apply when exporting logs. Only log entries that match the filter are exported." + default = null +} + +variable "description" { + type = string + description = "(Optional) A description of this sink. The maximum length of the description is 8000 characters." + default = null +} + +variable "disabled" { + type = bool + description = "(Optional) If set to True, then this sink is disabled and it does not export any log entries." + default = null +} + +variable "project" { + type = string + description = "(Optional) The ID of the project to create the sink in. If omitted, the project associated with the provider is used." + default = null +} + +variable "unique_writer_identity" { + type = bool + description = <<-END + (Optional) Whether or not to create a unique identity associated with this sink. + + If `false` (the default), then the `writer_identity` used is `serviceAccount:cloud-logs@system.gserviceaccount.com`. + + If `true`, then a unique service account is created and used for this sink. If you wish to publish logs across projects or utilize `bigquery_options`, you must set `unique_writer_identity` to true. + END + default = null +} + +# TODO: is this a good way for dealing with `bigquery_options` attribute? +variable "use_partitioned_tables" { + type = bool + description = <<-END + Whether to use BigQuery's partition tables. + + By default, Logging creates dated tables based on the log entries' timestamps, e.g. syslog_20170523. With partitioned tables the date suffix is no longer present and special query syntax has to be used instead. In both cases, tables are sharded based on UTC timezone. + END + default = null +} + +# variable "bigquery_options" { +# type = object({ +# use_partitioned_tables = bool +# }) # map(string)? any? +# description = "(Optional) Options that affect sinks exporting data to BigQuery." +# default = null +# } + +variable "exclusions" { + # type = list(object({ + # name = string + # filter = string + # description = optional(string) + # })) + type = any # TODO: is this correct? + + description = <<-END + Log entries that match any of the exclusion filters will not be exported. + + If a log entry is matched by both filter and one of `exclusion_filters` it will not be exported. Can be repeated multiple times for multiple exclusions. + END + default = [] +} + + # ---------------------------------------------------------------------------------------------------------------------- # MODULE CONFIGURATION PARAMETERS # These variables are used to configure the module. @@ -19,18 +113,6 @@ variable "module_enabled" { default = true } -# variable "module_tags" { -# type = map(string) -# description = "(Optional) A map of tags that will be applied to all created resources that accept tags. Tags defined with 'module_tags' can be overwritten by resource-specific tags." -# default = {} -# } - -# variable "module_timeouts" { -# description = "(Optional) A map of timeout objects that is keyed by Terraform resource name defining timeouts for `create`, `update` and `delete` Terraform operations." -# type = any -# default = null -# } - variable "module_depends_on" { type = any description = "(Optional) A list of external resources the module depends on." diff --git a/versions.tf b/versions.tf index 2ad10af..430d3d5 100644 --- a/versions.tf +++ b/versions.tf @@ -6,9 +6,9 @@ terraform { required_version = "~> 1.0" required_providers { - aws = { - source = "hashicorp/aws" - version = "~> 3.0" + google = { + source = "hashicorp/google" + version = "~> 4.1" } } }