[metricbeat] [gcp] add billing additional fields (elastic#36902)

* GCP Billing Enhancement Implemented a check to differentiate between standard and detailed cost usage tables Refactoring the 'row' structure used for storing BigQuery results, optimizing data handling Added support for tags Introduced new cost usage data fields * add tests fix older tests * add changelog entry * update docs and fields * add sku and service ids in hash * improve docs * update fields types * use array instead of map for tags include tags in the event signature * update sample doc * use sum instead of max for dashboards * fix label minor changes
Scholar-Li · Feb 5, 2024 · e0a6b0e · e0a6b0e
1 parent f06059b
commit e0a6b0e
Show file tree

Hide file tree

Showing 11 changed files with 498 additions and 45 deletions.
diff --git a/CHANGELOG.next.asciidoc b/CHANGELOG.next.asciidoc
@@ -281,6 +281,7 @@ is collected by it.
 - Add GCP Carbon Footprint metricbeat data {pull}34820[34820]
 - Add event loop utilization metric to Kibana module {pull}35020[35020]
 - Align on the algorithm used to transform Prometheus histograms into Elasticsearch histograms {pull}36647[36647]
+- Enhance GCP billing with detailed tables identification, additional fields, and optimized data handling. {pull}36902[36902]
 - Add a `/inputs/` route to the HTTP monitoring endpoint that exposes metrics for each metricset instance. {pull}36971[36971]
 
 

diff --git a/metricbeat/docs/fields.asciidoc b/metricbeat/docs/fields.asciidoc
@@ -35365,6 +35365,60 @@ type: float
 
 --
 
+*`gcp.billing.sku_id`*::
++
+--
+The ID of the resource used by the service.
+
+type: keyword
+
+--
+
+*`gcp.billing.sku_description`*::
++
+--
+A description of the resource type used by the service. For example, a resource type for Cloud Storage is Standard Storage US.
+
+type: keyword
+
+--
+
+*`gcp.billing.service_id`*::
++
+--
+The ID of the service that the usage is associated with.
+
+type: keyword
+
+--
+
+*`gcp.billing.service_description`*::
++
+--
+The Google Cloud service that reported the Cloud Billing data.
+
+type: keyword
+
+--
+
+*`gcp.billing.tags`*::
++
+--
+A collection of key-value pairs that provide additional metadata.
+
+type: nested
+
+--
+
+*`gcp.billing.effective_price`*::
++
+--
+The charged price for usage of the Google Cloud SKUs and SKU tiers. Reflects contract pricing if applicable, otherwise, it's the list price.
+
+type: float
+
+--
+
 [float]
 === carbon
 

diff --git a/x-pack/metricbeat/module/gcp/_meta/kibana/7/lens/057de170-e88d-11ea-bf8c-d13ebf358a78.json b/x-pack/metricbeat/module/gcp/_meta/kibana/7/lens/057de170-e88d-11ea-bf8c-d13ebf358a78.json
@@ -13,8 +13,8 @@
                                 "10b91492-efef-490d-bc7a-c2074b2eae84": {
                                     "dataType": "number",
                                     "isBucketed": false,
-                                    "label": "Maximum of gcp.billing.total",
-                                    "operationType": "max",
+                                    "label": "Sum of gcp.billing.total",
+                                    "operationType": "sum",
                                     "scale": "ratio",
                                     "sourceField": "gcp.billing.total"
                                 },

diff --git a/x-pack/metricbeat/module/gcp/_meta/kibana/7/lens/73346db0-e88d-11ea-bf8c-d13ebf358a78.json b/x-pack/metricbeat/module/gcp/_meta/kibana/7/lens/73346db0-e88d-11ea-bf8c-d13ebf358a78.json
@@ -16,7 +16,7 @@
                                     "dataType": "number",
                                     "isBucketed": false,
                                     "label": "Total Billing",
-                                    "operationType": "max",
+                                    "operationType": "sum",
                                     "scale": "ratio",
                                     "sourceField": "gcp.billing.total"
                                 },

diff --git a/x-pack/metricbeat/module/gcp/_meta/kibana/7/lens/e6933020-e88d-11ea-bf8c-d13ebf358a78.json b/x-pack/metricbeat/module/gcp/_meta/kibana/7/lens/e6933020-e88d-11ea-bf8c-d13ebf358a78.json
@@ -16,7 +16,7 @@
                                     "dataType": "number",
                                     "isBucketed": false,
                                     "label": "Total Billing Cost",
-                                    "operationType": "max",
+                                    "operationType": "sum",
                                     "scale": "ratio",
                                     "sourceField": "gcp.billing.total"
                                 },

diff --git a/x-pack/metricbeat/module/gcp/billing/_meta/data.json b/x-pack/metricbeat/module/gcp/billing/_meta/data.json
@@ -16,7 +16,18 @@
             "invoice_month": "202106",
             "project_id": "containerlib-prod-12763",
             "project_name": "elastic-containerlib-prod",
-            "total": 4717.170681
+            "total": 4717.170681,
+            "sku_id": "0D56-2F80-52A5",
+            "service_id": "6F81-5844-456A",
+            "sku_description": "Network Inter Region Ingress from Jakarta to Americas",
+            "service_description": "Compute Engine",
+            "effective_price": 0.00292353,
+            "tags": [
+                {
+                    "key": "size",
+                    "value": "standard"
+                }
+            ]
         }
     },
     "metricset": {

diff --git a/x-pack/metricbeat/module/gcp/billing/_meta/docs.asciidoc b/x-pack/metricbeat/module/gcp/billing/_meta/docs.asciidoc
@@ -1,18 +1,27 @@
 `billing` metricset is designed for collecting billing metrics from Google Cloud
 BigQuery daily cost detail table. BigQuery is a fully-managed, serverless data
 warehouse.
-Cloud Billing export to BigQuery enables you to export detailed Google Cloud
+Cloud Billing export to BigQuery enables you to export standard and detailed Google Cloud
 billing data (such as usage, cost estimates, and pricing data) automatically
 throughout the day to a BigQuery dataset that you specify. Then you can access
 your Cloud Billing data from BigQuery for detailed analysis using Metricbeat.
 Please see https://cloud.google.com/billing/docs/how-to/export-data-bigquery[export
 cloud billing data to BigQuery] for more details on how to export billing data.
 
-In BigQuery dataset, detailed Google Cloud daily cost data is loaded into a data
-table named `gcp_billing_export_v1_<BILLING_ACCOUNT_ID>`. There is a defined
-schema for Google Cloud daily cost data that is exported to BigQuery. Please see
-https://cloud.google.com/billing/docs/how-to/export-data-bigquery-tables#data-schema[
-daily cost detail data schema] for more details.
+In BigQuery, Google Cloud daily cost data is categorized into two formats:
+standard and detailed. Each format is stored within a designated dataset and
+follows a structured schema for precise cost analysis. For a comprehensive
+understanding of these formats, consult the https://cloud.google.com/billing/docs/how-to/export-data-bigquery-tables/standard-usage#standard-usage-cost-data-schema[
+standard] and https://cloud.google.com/billing/docs/how-to/export-data-bigquery-tables/detailed-usage#detailed-usage-cost-data-schema[
+detailed] data schema documentation.
+
+For standard usage cost data, set the table pattern format to
+`gcp_billing_export_v1`. This table pattern is set as the default when no other
+is specified.
+
+For detailed usage cost data, set the table pattern to `gcp_billing_export_resource_v1`.
+Detailed tables include the standard fields and additional fields, such as
+`effective_price`, enabling a more granular view of expenses.
 
 [float]
 === Metricset-specific configuration notes

diff --git a/x-pack/metricbeat/module/gcp/billing/_meta/fields.yml b/x-pack/metricbeat/module/gcp/billing/_meta/fields.yml
@@ -15,3 +15,26 @@
   - name: total
     type: float
     description: Total billing amount.
+  - name: sku_id
+    type: keyword
+    description: The ID of the resource used by the service.
+  - name: sku_description
+    type: keyword
+    description: A description of the resource type used by the service. For example, a resource type for Cloud Storage is Standard Storage US.
+  - name: service_id
+    type: keyword
+    description: The ID of the service that the usage is associated with.
+  - name: service_description
+    type: keyword
+    description: The Google Cloud service that reported the Cloud Billing data.
+  - name: tags
+    type: nested
+    description: A collection of key-value pairs that provide additional metadata.
+    fields:
+      - name: key
+        type: keyword
+      - name: value
+        type: keyword
+  - name: effective_price
+    type: float
+    description: The charged price for usage of the Google Cloud SKUs and SKU tiers. Reflects contract pricing if applicable, otherwise, it's the list price.