Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[GCP] Add billing additional fields #8343

Merged
merged 12 commits into from
Nov 22, 2023
+106 −10
Merged

[GCP] Add billing additional fields #8343

Show file tree
Hide file tree
Changes from 9 commits
Commits
Show all changes
12 commits
Select commit Hold shift + click to select a range
1303996
add additional fields
gpop63 Oct 30, 2023
c2b1f02
build docs
gpop63 Oct 30, 2023
ad386ef
use sum instead of max
gpop63 Oct 30, 2023
5c83775
bump package version
gpop63 Oct 31, 2023
1a81089
update sample event
gpop63 Nov 15, 2023
5d1d354
Merge remote-tracking branch 'upstream/main' into add_gcp_billing_fields
gpop63 Nov 15, 2023
5f579c2
Merge branch 'main' into add_gcp_billing_fields
gpop63 Nov 21, 2023
b1b35e8
Merge branch 'main' into add_gcp_billing_fields
gpop63 Nov 21, 2023
212d141
change label from max to sum
gpop63 Nov 21, 2023
d6b3f30
add tags in sample event
gpop63 Nov 22, 2023
7f5d426
add documentation
gpop63 Nov 22, 2023
aff9168
bump kibana version
gpop63 Nov 22, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions packages/gcp/changelog.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,9 @@
# newer versions go on top
- version: "2.32.0"
changes:
- description: Add new billing data stream fields.
type: enhancement
link: https://github.com/elastic/integrations/pull/8343
- version: "2.31.2"
changes:
- description: Fix pipeline error parsing DNS logs with empty rdata field.
Expand Down
24 changes: 24 additions & 0 deletions packages/gcp/data_stream/billing/fields/fields.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,3 +20,27 @@
- name: billing_account_id
type: keyword
description: Project Billing Account ID.
- name: sku_id
type: keyword
description: The ID of the resource used by the service.
- name: sku_description
type: keyword
description: A description of the resource type used by the service. For example, a resource type for Cloud Storage is Standard Storage US.
- name: service_id
type: keyword
description: The ID of the service that the usage is associated with.
- name: service_description
type: keyword
description: The Google Cloud service that reported the Cloud Billing data.
- name: effective_price
type: float
description: The charged price for usage of the Google Cloud SKUs and SKU tiers. Reflects contract pricing if applicable, otherwise, it's the list price.
- name: tags
type: nested
description: A collection of key-value pairs that provide additional metadata.
fields:
- name: key
type: keyword
- name: value
type: keyword

8 changes: 7 additions & 1 deletion packages/gcp/data_stream/billing/sample_event.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,13 @@
"invoice_month": "202106",
"project_id": "containerlib-prod-12763",
"project_name": "elastic-containerlib-prod",
"total": 4717.170681
"total": 4717.170681,
"sku_id": "0D56-2F80-52A5",
"service_id": "6F81-5844-456A",
"sku_description": "Network Inter Region Ingress from Jakarta to Americas",
"service_description": "Compute Engine",
"effective_price": 0.00292353,
"tags": {}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: It would be helpful to include tag values in the sample event.

}
},
"metricset": {
Expand Down
15 changes: 14 additions & 1 deletion packages/gcp/docs/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1493,9 +1493,16 @@ The `billing` dataset collects GCP Billing information from Google Cloud BigQuer
| event.module | Event module | constant_keyword |
| gcp.billing.billing_account_id | Project Billing Account ID. | keyword |
| gcp.billing.cost_type | Cost types include regular, tax, adjustment, and rounding_error. | keyword |
| gcp.billing.effective_price | The charged price for usage of the Google Cloud SKUs and SKU tiers. Reflects contract pricing if applicable, otherwise, it's the list price. | float |
| gcp.billing.invoice_month | Billing report month. | keyword |
| gcp.billing.project_id | Project ID of the billing report belongs to. | keyword |
| gcp.billing.project_name | Project Name of the billing report belongs to. | keyword |
| gcp.billing.service_description | The Google Cloud service that reported the Cloud Billing data. | keyword |
| gcp.billing.service_id | The ID of the service that the usage is associated with. | keyword |
| gcp.billing.sku_description | A description of the resource type used by the service. For example, a resource type for Cloud Storage is Standard Storage US. | keyword |
| gcp.billing.sku_id | The ID of the resource used by the service. | keyword |
| gcp.billing.tags.key | | keyword |
| gcp.billing.tags.value | | keyword |
| gcp.billing.total | Total billing amount. | float |
| host.architecture | Operating system architecture. | keyword |
| host.containerized | If the host is a container. | boolean |
Expand Down Expand Up @@ -1544,7 +1551,13 @@ An example event for `billing` looks as following:
"invoice_month": "202106",
"project_id": "containerlib-prod-12763",
"project_name": "elastic-containerlib-prod",
"total": 4717.170681
"total": 4717.170681,
"sku_id": "0D56-2F80-52A5",
"service_id": "6F81-5844-456A",
"sku_description": "Network Inter Region Ingress from Jakarta to Americas",
"service_description": "Compute Engine",
"effective_price": 0.00292353,
"tags": {}
}
},
"metricset": {
Expand Down
15 changes: 14 additions & 1 deletion packages/gcp/docs/billing.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -37,7 +37,13 @@ An example event for `billing` looks as following:
"invoice_month": "202106",
"project_id": "containerlib-prod-12763",
"project_name": "elastic-containerlib-prod",
"total": 4717.170681
"total": 4717.170681,
"sku_id": "0D56-2F80-52A5",
"service_id": "6F81-5844-456A",
"sku_description": "Network Inter Region Ingress from Jakarta to Americas",
"service_description": "Compute Engine",
"effective_price": 0.00292353,
"tags": {}
}
},
"metricset": {
Expand Down Expand Up @@ -82,9 +88,16 @@ An example event for `billing` looks as following:
| event.module | Event module | constant_keyword |
| gcp.billing.billing_account_id | Project Billing Account ID. | keyword |
| gcp.billing.cost_type | Cost types include regular, tax, adjustment, and rounding_error. | keyword |
| gcp.billing.effective_price | The charged price for usage of the Google Cloud SKUs and SKU tiers. Reflects contract pricing if applicable, otherwise, it's the list price. | float |
| gcp.billing.invoice_month | Billing report month. | keyword |
| gcp.billing.project_id | Project ID of the billing report belongs to. | keyword |
| gcp.billing.project_name | Project Name of the billing report belongs to. | keyword |
| gcp.billing.service_description | The Google Cloud service that reported the Cloud Billing data. | keyword |
| gcp.billing.service_id | The ID of the service that the usage is associated with. | keyword |
| gcp.billing.sku_description | A description of the resource type used by the service. For example, a resource type for Cloud Storage is Standard Storage US. | keyword |
| gcp.billing.sku_id | The ID of the resource used by the service. | keyword |
| gcp.billing.tags.key | | keyword |
| gcp.billing.tags.value | | keyword |
| gcp.billing.total | Total billing amount. | float |
| host.architecture | Operating system architecture. | keyword |
| host.containerized | If the host is a container. | boolean |
Expand Down
8 changes: 4 additions & 4 deletions packages/gcp/kibana/dashboard/gcp-76c9e920-e890-11ea-bf8c-d13ebf358a78.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -143,8 +143,8 @@
"10b91492-efef-490d-bc7a-c2074b2eae84": {
"dataType": "number",
"isBucketed": false,
"label": "Maximum of gcp.billing.total",
"operationType": "max",
"label": "Sum of gcp.billing.total",
"operationType": "sum",
"scale": "ratio",
"sourceField": "gcp.billing.total"
},
Expand Down Expand Up @@ -337,7 +337,7 @@
"dataType": "number",
"isBucketed": false,
"label": "Total Billing Cost",
"operationType": "max",
"operationType": "sum",
"scale": "ratio",
"sourceField": "gcp.billing.total"
},
Expand Down Expand Up @@ -459,7 +459,7 @@
"dataType": "number",
"isBucketed": false,
"label": "Total Billing",
"operationType": "max",
"operationType": "sum",
"scale": "ratio",
"sourceField": "gcp.billing.total"
},
Expand Down
2 changes: 1 addition & 1 deletion packages/gcp/manifest.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
name: gcp
title: Google Cloud Platform
version: "2.31.2"
version: "2.32.0"
description: Collect logs and metrics from Google Cloud Platform with Elastic Agent.
type: integration
icons:
Expand Down