Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Allow custom labels #165

Merged
merged 21 commits into from
May 28, 2024
Merged

Conversation

dj707chen
Copy link
Contributor

@dj707chen dj707chen commented May 23, 2024

Allow adding custom labels.

Background
In a situation where a service interacts with multiple providers, a few of them do not have a lot of traffic, so when one of these light providers is down, the alert of the service is not triggered due to the small percentage counted in the failing traffic; So we want to create alerts specific to providers, and I’m searching for ways to identify the metrics entries related to a provider.
Two options currently available:

  • Use provider specific prefix when create MetricsOps instance, but the metrics of this client will be excluded from a generic Http4S metrics dashboard;
  • Add provider specific prefix to classifier when create org.http4s.client.middleware.Metrics.

The second option is better than the first option, but not ideal.

What it does
This PR enables clients to pass custom labels and corresponding values when create MetricsOps and allow alerts targeting those label.

Code refactor
There were a few configuration items (prefix, registry, sampleExemplar, responseDurationSecondsHistogramBuckets), to reduce the number of methods, converted the code to builder pattern, then added withCustomLabelsAndValues method.

Example
Here is an example. If we create a MetricsOps as below and pass it to a client Metrics middleware

  val metricsOpsPaypal = Prometheus.default(registry)
        .withCustomLabelsAndValues(List(("provider", "Paypal"))
        .build
  val paypalClient = BlazeClientBuilder[F]
      .resource
      .map { client => org.http4s.client.middleware.Metrics[F](metricsOpsPaypal)(client) }

Now we can create a Prometheus rule as below, note, label status is set by Metrics middleware, app by filter, and provider is a custom label.

- name: ./team1.rules
   rules:
   - alert: PaypalProviderFailed
      expr: |
         100 * (
             sum by(classifier)(rate(http4s:org_http4s_server_request_count_total{app="banking", provider="Paypal", status="5xx",}[2m]))
             /
             sum by(classifier)(rate(http4s:org_http4s_server_request_count_total{app="banking", provider="Paypal"}[2m]))
         ) > 5
      for: 5m
      labels:
         severity: warning
         team: "team-1"
   - alert: ZelleProviderFailed
      ......

@dj707chen dj707chen requested a review from rossabaker May 25, 2024 18:00
Copy link
Member

@rossabaker rossabaker left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good, thanks!

@rossabaker rossabaker merged commit cffd5f7 into http4s:series/0.24 May 28, 2024
9 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants