Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Support extra label selectors on mimir.rules.kubernetes #1773

Merged
merged 5 commits into from
Sep 30, 2024
+447 −119
Merged

Support extra label selectors on mimir.rules.kubernetes #1773

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
7c57d39
support extra labels on mimir rules queries
thampiotr Sep 26, 2024
c3b0d96
Add tests and expose query matchers to users.
thampiotr Sep 27, 2024
04d6303
Docs
thampiotr Sep 27, 2024
453e253
Changelog
thampiotr Sep 27, 2024
a13c540
Apply suggestions from code review
thampiotr Sep 30, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions CHANGELOG.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,11 @@ Main (unreleased)

- Add the function `path_join` to the stdlib. (@wildum)

### Enhancements

- The `mimir.rules.kubernetes` component now supports adding extra label matchers
to all queries discovered via `PrometheusRule` CRDs. (@thampiotr)

### Bugfixes

- Windows installer: Don't quote alloy binary path in registy (@jkroepke)
Expand Down
132 changes: 87 additions & 45 deletions docs/sources/reference/components/mimir/mimir.rules.kubernetes.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -50,25 +50,25 @@ mimir.rules.kubernetes "LABEL" {

`mimir.rules.kubernetes` supports the following arguments:

Name | Type | Description | Default | Required
-------------------------|---------------------|--------------------------------------------------------------------------------------------------|---------------|---------
`address` | `string` | URL of the Mimir ruler. | | yes
`tenant_id` | `string` | Mimir tenant ID. | | no
`use_legacy_routes` | `bool` | Whether to use [deprecated][gem-2_2] ruler API endpoints. | false | no
`prometheus_http_prefix` | `string` | Path prefix for [Mimir's Prometheus endpoint][gem-path-prefix]. | `/prometheus` | no
`sync_interval` | `duration` | Amount of time between reconciliations with Mimir. | "5m" | no
`mimir_namespace_prefix` | `string` | Prefix used to differentiate multiple {{< param "PRODUCT_NAME" >}} deployments. | "alloy" | no
`bearer_token_file` | `string` | File containing a bearer token to authenticate with. | | no
`bearer_token` | `secret` | Bearer token to authenticate with. | | no
`enable_http2` | `bool` | Whether HTTP2 is supported for requests. | `true` | no
`follow_redirects` | `bool` | Whether redirects returned by the server should be followed. | `true` | no
`proxy_url` | `string` | HTTP proxy to send requests through. | | no
`no_proxy` | `string` | Comma-separated list of IP addresses, CIDR notations, and domain names to exclude from proxying. | | no
`proxy_from_environment` | `bool` | Use the proxy URL indicated by environment variables. | `false` | no
`proxy_connect_header` | `map(list(secret))` | Specifies headers to send to proxies during CONNECT requests. | | no
`external_labels` | `map(string)` | Labels to add to each rule. | `{}` | no

At most, one of the following can be provided:
| Name | Type | Description | Default | Required |
|--------------------------|---------------------|--------------------------------------------------------------------------------------------------|---------------|----------|
| `address` | `string` | URL of the Mimir ruler. | | yes |
| `tenant_id` | `string` | Mimir tenant ID. | | no |
| `use_legacy_routes` | `bool` | Whether to use [deprecated][gem-2_2] ruler API endpoints. | false | no |
| `prometheus_http_prefix` | `string` | Path prefix for [Mimir's Prometheus endpoint][gem-path-prefix]. | `/prometheus` | no |
| `sync_interval` | `duration` | Amount of time between reconciliations with Mimir. | "5m" | no |
| `mimir_namespace_prefix` | `string` | Prefix used to differentiate multiple {{< param "PRODUCT_NAME" >}} deployments. | "alloy" | no |
| `bearer_token_file` | `string` | File containing a bearer token to authenticate with. | | no |
| `bearer_token` | `secret` | Bearer token to authenticate with. | | no |
| `enable_http2` | `bool` | Whether HTTP2 is supported for requests. | `true` | no |
| `follow_redirects` | `bool` | Whether redirects returned by the server should be followed. | `true` | no |
| `proxy_url` | `string` | HTTP proxy to send requests through. | | no |
| `no_proxy` | `string` | Comma-separated list of IP addresses, CIDR notations, and domain names to exclude from proxying. | | no |
| `proxy_from_environment` | `bool` | Use the proxy URL indicated by environment variables. | `false` | no |
| `proxy_connect_header` | `map(list(secret))` | Specifies headers to send to proxies during CONNECT requests. | | no |
| `external_labels` | `map(string)` | Labels to add to each rule. | `{}` | no |

At most, one of the following can be provided:
- [`bearer_token` argument](#arguments).
- [`bearer_token_file` argument](#arguments).
- [`basic_auth` block][basic_auth].
Expand Down Expand Up @@ -105,17 +105,19 @@ This is useful if you configure Mimir to use a different [prefix][gem-path-prefi
The following blocks are supported inside the definition of
`mimir.rules.kubernetes`:

Hierarchy | Block | Description | Required
-------------------------------------------|------------------------|----------------------------------------------------------|---------
rule_namespace_selector | [label_selector][] | Label selector for `Namespace` resources. | no
rule_namespace_selector > match_expression | [match_expression][] | Label match expression for `Namespace` resources. | no
rule_selector | [label_selector][] | Label selector for `PrometheusRule` resources. | no
rule_selector > match_expression | [match_expression][] | Label match expression for `PrometheusRule` resources. | no
basic_auth | [basic_auth][] | Configure basic_auth for authenticating to the endpoint. | no
authorization | [authorization][] | Configure generic authorization to the endpoint. | no
oauth2 | [oauth2][] | Configure OAuth2 for authenticating to the endpoint. | no
oauth2 > tls_config | [tls_config][] | Configure TLS settings for connecting to the endpoint. | no
tls_config | [tls_config][] | Configure TLS settings for connecting to the endpoint. | no
| Hierarchy | Block | Description | Required |
|--------------------------------------------|--------------------------|----------------------------------------------------------|----------|
| extra_query_matchers | [extra_query_matchers][] | Additional label matchers to add to each query. | no |
| extra_query_matchers > matcher | [matcher][] | A label matcher to add to query. | no |
| rule_namespace_selector | [label_selector][] | Label selector for `Namespace` resources. | no |
| rule_namespace_selector > match_expression | [match_expression][] | Label match expression for `Namespace` resources. | no |
| rule_selector | [label_selector][] | Label selector for `PrometheusRule` resources. | no |
| rule_selector > match_expression | [match_expression][] | Label match expression for `PrometheusRule` resources. | no |
| basic_auth | [basic_auth][] | Configure basic_auth for authenticating to the endpoint. | no |
| authorization | [authorization][] | Configure generic authorization to the endpoint. | no |
| oauth2 | [oauth2][] | Configure OAuth2 for authenticating to the endpoint. | no |
| oauth2 > tls_config | [tls_config][] | Configure TLS settings for connecting to the endpoint. | no |
| tls_config | [tls_config][] | Configure TLS settings for connecting to the endpoint. | no |

The `>` symbol indicates deeper levels of nesting. For example,
`oauth2 > tls_config` refers to a `tls_config` block defined inside
Expand All @@ -127,16 +129,38 @@ an `oauth2` block.
[tls_config]: #tls_config-block
[label_selector]: #label_selector-block
[match_expression]: #match_expression-block
[extra_query_matchers]: #extra_query_matchers-block
[matcher]: #matcher-block

### extra_query_matchers block

The `extra_query_matchers` block has no attributes. It contains zero or more [matcher][] blocks.
These blocks allow you to add extra label matchers to all queries that are discovered by `mimir.rules.kubernetes`
component. The algorithm of adding the label matchers to queries is the same as the one provided by
`promtool promql label-matchers set` command in [promtool](https://github.com/prometheus/prometheus/tree/main/cmd/promtool).

### matcher block

The `matcher` block describes a label matcher that will be added to each query found in `PrometheusRule` CRDs.

The following arguments are supported:

| Name | Type | Description | Default | Required |
|--------------|----------|----------------------------------------------------|---------|----------|
| `name` | `string` | Name of the label to match. | | yes |
wildum marked this conversation as resolved.
Show resolved Hide resolved
| `match_type` | `string` | The type of match. One of `=`, `!=`, `=~` and `!~`. | | yes |
| `value` | `string` | Value of the label to match. | | yes |


### label_selector block

The `label_selector` block describes a Kubernetes label selector for rule or namespace discovery.

The following arguments are supported:

Name | Type | Description | Default | Required
---------------|---------------|---------------------------------------------------|-----------------------------|---------
`match_labels` | `map(string)` | Label keys and values used to discover resources. | `{}` | yes
| Name | Type | Description | Default | Required |
|----------------|---------------|---------------------------------------------------|---------|----------|
| `match_labels` | `map(string)` | Label keys and values used to discover resources. | `{}` | yes |

When the `match_labels` argument is empty, all resources will be matched.

Expand All @@ -146,11 +170,11 @@ The `match_expression` block describes a Kubernetes label match expression for r

The following arguments are supported:

Name | Type | Description | Default | Required
-----------|----------------|----------------------------------------------------|---------|---------
`key` | `string` | The label name to match against. | | yes
`operator` | `string` | The operator to use when matching. | | yes
`values` | `list(string)` | The values used when matching. | | no
| Name | Type | Description | Default | Required |
|------------|----------------|------------------------------------|---------|----------|
| `key` | `string` | The label name to match against. | | yes |
| `operator` | `string` | The operator to use when matching. | | yes |
| `values` | `list(string)` | The values used when matching. | | no |

The `operator` argument should be one of the following strings:

Expand Down Expand Up @@ -204,13 +228,13 @@ actually exist.

## Debug metrics

Metric Name | Type | Description
----------------------------------------------|-------------|-------------------------------------------------------------------------
`mimir_rules_config_updates_total` | `counter` | Number of times the configuration has been updated.
`mimir_rules_events_total` | `counter` | Number of events processed, partitioned by event type.
`mimir_rules_events_failed_total` | `counter` | Number of events that failed to be processed, partitioned by event type.
`mimir_rules_events_retried_total` | `counter` | Number of events that were retried, partitioned by event type.
`mimir_rules_client_request_duration_seconds` | `histogram` | Duration of requests to the Mimir API.
| Metric Name | Type | Description |
|-----------------------------------------------|-------------|--------------------------------------------------------------------------|
| `mimir_rules_config_updates_total` | `counter` | Number of times the configuration has been updated. |
| `mimir_rules_events_total` | `counter` | Number of events processed, partitioned by event type. |
| `mimir_rules_events_failed_total` | `counter` | Number of events that failed to be processed, partitioned by event type. |
| `mimir_rules_events_retried_total` | `counter` | Number of events that were retried, partitioned by event type. |
| `mimir_rules_client_request_duration_seconds` | `histogram` | Duration of requests to the Mimir API. |

## Example

Expand Down Expand Up @@ -253,6 +277,24 @@ mimir.rules.kubernetes "default" {
}
```

This example adds label matcher `{cluster=~"prod-.*"}` to all the queries discovered by `mimir.rules.kubernetes`.

```alloy
mimir.rules.kubernetes "default" {
address = "GRAFANA_CLOUD_METRICS_URL"
extra_query_matchers {
matcher {
name = "cluster"
match_type = "=~"
value = "prod-.*"
}
}
}
```

If a query in the form of `up != 1` is found in `PrometheusRule` CRDs,
it will be modified to `up{cluster=~"prod-.*"} != 1` before sending it to Mimir.

The following example is an RBAC configuration for Kubernetes. It authorizes {{< param "PRODUCT_NAME" >}} to query the Kubernetes REST API:

```yaml
Expand Down
Loading
Loading