elastic · gpop63 · Nov 3, 2023 · Oct 22, 2023 · Oct 22, 2023 · Oct 22, 2023
diff --git a/CHANGELOG.next.asciidoc b/CHANGELOG.next.asciidoc
@@ -271,6 +271,7 @@ is collected by it.
 - Add GCP Carbon Footprint metricbeat data {pull}34820[34820]
 - Add event loop utilization metric to Kibana module {pull}35020[35020]
 - Align on the algorithm used to transform Prometheus histograms into Elasticsearch histograms {pull}36647[36647]
+- Enhance GCP billing with detailed tables identification, additional fields, and optimized data handling. {pull}36902[36902]
 - Add a `/inputs/` route to the HTTP monitoring endpoint that exposes metrics for each metricset instance. {pull}36971[36971]
 
 

diff --git a/metricbeat/docs/fields.asciidoc b/metricbeat/docs/fields.asciidoc
@@ -35365,6 +35365,60 @@ type: float
 
 --
 
+*`gcp.billing.sku_id`*::
++
+--
+The ID of the resource used by the service.
+
+type: keyword
+
+--
+
+*`gcp.billing.sku_description`*::
++
+--
+A description of the resource type used by the service. For example, a resource type for Cloud Storage is Standard Storage US.
+
+type: keyword
+
+--
+
+*`gcp.billing.service_id`*::
++
+--
+The ID of the service that the usage is associated with.
+
+type: keyword
+
+--
+
+*`gcp.billing.service_description`*::
++
+--
+The Google Cloud service that reported the Cloud Billing data.
+
+type: keyword
+
+--
+
+*`gcp.billing.tags`*::
++
+--
+A collection of key-value pairs that provide additional metadata.
+
+type: nested
+
+--
+
+*`gcp.billing.effective_price`*::
++
+--
+The charged price for usage of the Google Cloud SKUs and SKU tiers. Reflects contract pricing if applicable, otherwise, it's the list price.
+
+type: float
+
+--
+
 [float]
 === carbon
 

@@ -13,8 +13,8 @@
                                 "10b91492-efef-490d-bc7a-c2074b2eae84": {
                                     "dataType": "number",
                                     "isBucketed": false,
-                                    "label": "Maximum of gcp.billing.total",
-                                    "operationType": "max",
+                                    "label": "Sum of gcp.billing.total",
+                                    "operationType": "sum",
                                     "scale": "ratio",
                                     "sourceField": "gcp.billing.total"
                                 },

@@ -16,7 +16,7 @@
                                     "dataType": "number",
                                     "isBucketed": false,
                                     "label": "Total Billing",
-                                    "operationType": "max",
+                                    "operationType": "sum",
                                     "scale": "ratio",
                                     "sourceField": "gcp.billing.total"
                                 },

@@ -16,7 +16,7 @@
                                     "dataType": "number",
                                     "isBucketed": false,
                                     "label": "Total Billing Cost",
-                                    "operationType": "max",
+                                    "operationType": "sum",
                                     "scale": "ratio",
                                     "sourceField": "gcp.billing.total"
                                 },

@@ -16,7 +16,18 @@
             "invoice_month": "202106",
             "project_id": "containerlib-prod-12763",
             "project_name": "elastic-containerlib-prod",
-            "total": 4717.170681
+            "total": 4717.170681,
+            "sku_id": "0D56-2F80-52A5",
+            "service_id": "6F81-5844-456A",
+            "sku_description": "Network Inter Region Ingress from Jakarta to Americas",
+            "service_description": "Compute Engine",
+            "effective_price": 0.00292353,
+            "tags": [
+                {
+                    "key": "size",
+                    "value": "standard"
+                }
+            ]
         }
     },
     "metricset": {

@@ -1,18 +1,27 @@
 `billing` metricset is designed for collecting billing metrics from Google Cloud
 BigQuery daily cost detail table. BigQuery is a fully-managed, serverless data
 warehouse.
-Cloud Billing export to BigQuery enables you to export detailed Google Cloud
+Cloud Billing export to BigQuery enables you to export standard and detailed Google Cloud
 billing data (such as usage, cost estimates, and pricing data) automatically
 throughout the day to a BigQuery dataset that you specify. Then you can access
 your Cloud Billing data from BigQuery for detailed analysis using Metricbeat.
 Please see https://cloud.google.com/billing/docs/how-to/export-data-bigquery[export
 cloud billing data to BigQuery] for more details on how to export billing data.
 
-In BigQuery dataset, detailed Google Cloud daily cost data is loaded into a data
-table named `gcp_billing_export_v1_<BILLING_ACCOUNT_ID>`. There is a defined
-schema for Google Cloud daily cost data that is exported to BigQuery. Please see
-https://cloud.google.com/billing/docs/how-to/export-data-bigquery-tables#data-schema[
-daily cost detail data schema] for more details.
+In BigQuery, Google Cloud daily cost data is categorized into two formats:
+standard and detailed. Each format is stored within a designated dataset and
+follows a structured schema for precise cost analysis. For a comprehensive
+understanding of these formats, consult the https://cloud.google.com/billing/docs/how-to/export-data-bigquery-tables/standard-usage#standard-usage-cost-data-schema[
+standard] and https://cloud.google.com/billing/docs/how-to/export-data-bigquery-tables/detailed-usage#detailed-usage-cost-data-schema[
+detailed] data schema documentation.
+
+For standard usage cost data, set the table pattern format to
+`gcp_billing_export_v1`. This table pattern is set as the default when no other
+is specified.
+
+For detailed usage cost data, set the table pattern to `gcp_billing_export_resource_v1`.
+Detailed tables include the standard fields and additional fields, such as
+`effective_price`, enabling a more granular view of expenses.
 
 [float]
 === Metricset-specific configuration notes

@@ -15,3 +15,26 @@
   - name: total
     type: float
     description: Total billing amount.
+  - name: sku_id
+    type: keyword
+    description: The ID of the resource used by the service.
+  - name: sku_description
+    type: keyword
+    description: A description of the resource type used by the service. For example, a resource type for Cloud Storage is Standard Storage US.
+  - name: service_id
+    type: keyword
+    description: The ID of the service that the usage is associated with.
+  - name: service_description
+    type: keyword
+    description: The Google Cloud service that reported the Cloud Billing data.
+  - name: tags
+    type: nested
+    description: A collection of key-value pairs that provide additional metadata.
+    fields:
+      - name: key
+        type: keyword
+      - name: value
+        type: keyword
+  - name: effective_price
+    type: float
+    description: The charged price for usage of the Google Cloud SKUs and SKU tiers. Reflects contract pricing if applicable, otherwise, it's the list price.