Skip to content

Commit

Permalink
Replace event.metric_names with metric_names_hash
Browse files Browse the repository at this point in the history
A single GCP metric name can be quite long, for example:

  subscription.streaming_pull_mod_ack_deadline_message_operation.count

we may collect enough metrics to overflow the current
length limit for dimension fields.

We hash the metric names value using SHA-256 to have a constant length
value.

I'm not 100% sure SHA-256 is the best option for this use case; we
don't have cryptographic solid needs, so we can probably use a
simpler algorithm to save computing cycles while getting a shorter hash
value.

We also drop `event.metric_names` because is no longer needed.
  • Loading branch information
zmoog committed Oct 19, 2023
1 parent 56799ed commit efe6a55
Show file tree
Hide file tree
Showing 5 changed files with 23 additions and 9 deletions.
4 changes: 2 additions & 2 deletions metricbeat/docs/fields.asciidoc
Original file line number Diff line number Diff line change
Expand Up @@ -35303,10 +35303,10 @@ GCP module



*`event.metric_names`*::
*`event.metric_names_hash`*::
+
--
The comma-separated list metric names collected in the batch. For example, l3.external.ingress_packets.count,l3.external.ingress.bytes. Required to support TSDB.
The SHA-256 hash of the comma-separated list metric names collected in the batch. For example, l3.external.ingress_packets.count,l3.external.ingress.bytes > 24848afd85b65b9e168fa5dee12bd3e121c58a504cc0b4386a15bd9bcd065a5d. Required to support TSDB.


type: keyword
Expand Down
4 changes: 2 additions & 2 deletions x-pack/metricbeat/module/gcp/_meta/fields.yml
Original file line number Diff line number Diff line change
Expand Up @@ -7,10 +7,10 @@
- name: event
type: group
fields:
- name: metric_names
- name: metric_names_hash
type: keyword
description: >
The comma-separated list metric names collected in the batch. For example, l3.external.ingress_packets.count,l3.external.ingress.bytes. Required to support TSDB.
The SHA-256 hash of the comma-separated list metric names collected in the batch. For example, l3.external.ingress_packets.count,l3.external.ingress.bytes > 24848afd85b65b9e168fa5dee12bd3e121c58a504cc0b4386a15bd9bcd065a5d. Required to support TSDB.
- name: gcp
type: group
fields:
Expand Down
2 changes: 1 addition & 1 deletion x-pack/metricbeat/module/gcp/fields.go

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

16 changes: 15 additions & 1 deletion x-pack/metricbeat/module/gcp/metrics/timeseries.go
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,8 @@ package metrics

import (
"context"
"crypto/sha256"
"encoding/hex"
"fmt"
"strings"

Expand Down Expand Up @@ -140,7 +142,12 @@ func createEventsFromGroups(service string, groups map[string][]KeyValuePoint) [
event.RootFields = group[0].ECS
}

_, _ = event.RootFields.Put("event.metric_names", strings.Join(metricNames, ","))
// Hashes metric names string using SHA-256 to always have
// a constant length value and avoid overflowing the
// current TSDB dimension field limit (1024).
metricNamesHash := hash(strings.Join(metricNames, ","))

_, _ = event.RootFields.Put("event.metric_names_hash", metricNamesHash)

events = append(events, event)
}
Expand Down Expand Up @@ -188,3 +195,10 @@ func (m *MetricSet) groupTimeSeries(ctx context.Context, timeSeries []timeSeries

return groupedMetrics
}

// hash return the SHA-256 hash of the input string.
func hash(s string) string {
h := sha256.New()
h.Write([]byte(s))
return hex.EncodeToString(h.Sum(nil))
}
6 changes: 3 additions & 3 deletions x-pack/metricbeat/module/gcp/metrics/timeseries_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -475,7 +475,7 @@ func TestCreateEventsFromGroup(t *testing.T) {
"cloud.provider": "gcp",
"cloud.region": "us-west",
"event": mapstr.M{
"metric_names": "metric1,metric2",
"metric_names_hash": "2a990d0ce61c177d3bc05def9f536de164d3ea10056b7120414ad370a0f686c0",
},
},
},
Expand Down Expand Up @@ -504,7 +504,7 @@ func TestCreateEventsFromGroup(t *testing.T) {
"cloud.provider": "gcp",
"cloud.region": "us-east",
"event": mapstr.M{
"metric_names": "metric3,metric4",
"metric_names_hash": "f7c8d8a9a75411effbd2171a8fa984a5e618eeca3911829b03c7e3e3d6e74522",
},
},
},
Expand Down Expand Up @@ -532,7 +532,7 @@ func TestCreateEventsFromGroup(t *testing.T) {
"cloud.provider": "gcp",
"cloud.region": "us-east",
"event": mapstr.M{
"metric_names": "metric5",
"metric_names_hash": "ce7dee36d9ab56de52176bf05d21c0bc446da9a3134727bf5e300604f2ca63ef",
},
},
},
Expand Down

0 comments on commit efe6a55

Please sign in to comment.