diff --git a/docs/sources/tempo/introduction/solutions-with-traces/traces-app-insights.md b/docs/sources/tempo/introduction/solutions-with-traces/traces-app-insights.md index ce35b6fffb2..e912780367d 100644 --- a/docs/sources/tempo/introduction/solutions-with-traces/traces-app-insights.md +++ b/docs/sources/tempo/introduction/solutions-with-traces/traces-app-insights.md @@ -7,6 +7,17 @@ keywords: title: Identify bottlenecks and establish SLOs menuTitle: Identify bottlenecks and establish SLOs weight: 320 +refs: + metrics-generator: + - pattern: /docs/tempo/ + destination: https://grafana.com/docs/tempo//metrics-generator/ + - pattern: /docs/enterprise-traces/ + destination: https://grafana.com/docs/enterprise-traces//metrics-generator/ + span-metrics: + - pattern: /docs/tempo/ + destination: https://grafana.com/docs/tempo//metrics-generator/span_metrics/ + - pattern: /docs/enterprise-traces/ + destination: https://grafana.com/docs/enterprise-traces//metrics-generator/span_metrics/ --- # Identify bottlenecks and establish SLOs @@ -19,34 +30,34 @@ Handy Site Corp, a fake website company, runs an ecommerce application that incl ### Define realistic SLOs -Handy Site’s engineers start by establishing service level objectives (SLOs) around latency ensure that customers have a good experience when trying to complete the checkout process. +Handy Site’s engineers start by establishing service level objectives (SLOs) around latency ensure that customers have a good experience when trying to complete the checkout process. To do this, they use metrics generated from their span data. Their service level objective should be a realistic target based on previous history during times of normal operation. -Once they've agreed upon their service level objective, they will set up alerts to warn them when they are at risk of failing to meet that objective. +Once they've agreed upon their service level objective, they will set up alerts to warn them when they are at risk of failing to meet that objective. ### Utilize span metrics to define your SLO and SLI -After evaluating options, they decide to use [span metrics](https://grafana.com/docs/tempo/latest/metrics-generator/span_metrics/) as a service level indicator (SLI) to measure SLO compliance. +After evaluating options, they decide to use [span metrics](ref:span-metrics) as a service level indicator (SLI) to measure SLO compliance. ![Metrics generator and exemplars](/media/docs/tempo/intro/traces-metrics-gen-exemplars.png) -Tempo can generate metrics using the [metrics-generator component](https://grafana.com/docs/tempo/latest/metrics-generator/). +Tempo can generate metrics using the [metrics-generator component](ref:metrics-generator). These metrics are created based on spans from incoming traces and demonstrate immediate usefulness with respect to application flow and overview. This includes rate, error, and duration (RED) metrics. Span metrics also make it easy to use exemplars. -An [exemplar](https://grafana.com/docs/grafana/latest/basics/exemplars/) serves as a detailed example of one of the observations aggregated into a metric. An exemplar contains the observed value together with an optional timestamp and arbitrary trace IDs, which are typically used to reference a trace. -Since traces and metrics co-exist in the metrics-generator, exemplars can be automatically added to those metrics, allowing you to quickly jump from a metric showing aggregate latency over time into an individual trace that represents a low, medium, or high latency request. Similarly, you can quickly jump from a metric showing error rate over time into an individual erroring trace. +An [exemplar](https://grafana.com/docs/grafana//basics/exemplars/) serves as a detailed example of one of the observations aggregated into a metric. An exemplar contains the observed value together with an optional timestamp and arbitrary trace IDs, which are typically used to reference a trace. +Since traces and metrics co-exist in the metrics-generator, exemplars can be automatically added to those metrics, allowing you to quickly jump from a metric showing aggregate latency over time into an individual trace that represents a low, medium, or high latency request. Similarly, you can quickly jump from a metric showing error rate over time into an individual erroring trace. ### Monitor latency Handy Site decides they're most interested in monitoring the latency of requests processed by their checkout service and want to set an objective that 99.5% of requests in a given month should complete within 2 seconds. To define a service level indicator (SLI) that they can use to track their progress against their objective, they use the `traces_spanmetrics_latency` metric with the proper label selectors, such as `service name = checkoutservice`. -The metrics-generator adds a default set of labels to the metrics it generates, including `span_kind` and `status_code`. However, if they were interested in calculating checkout service latency per endpoint or per version of the software, they could change the configuration of the Tempo metrics-generator to add these custom dimensions as labels to their spanmetrics. +The metrics-generator adds a default set of labels to the metrics it generates, including `span_kind` and `status_code`. However, if they were interested in calculating checkout service latency per endpoint or per version of the software, they could change the configuration of the Tempo metrics-generator to add these custom dimensions as labels to their spanmetrics. -With all of this in place, Handy Site now opens the [Grafana SLO](https://grafana.com/docs/grafana-cloud/alerting-and-irm/slo/) application and follows the setup flow to establish an [SLI](https://grafana.com/docs/grafana-cloud/alerting-and-irm/slo/create/) for their checkout service around the `traces_spanmetrics_latency` metric.. +With all of this in place, Handy Site now opens the [Grafana SLO](https://grafana.com/docs/grafana-cloud/alerting-and-irm/slo/) application and follows the setup flow to establish an [SLI](https://grafana.com/docs/grafana-cloud/alerting-and-irm/slo/create/) for their checkout service around the `traces_spanmetrics_latency` metric. They can now be alerted to degradations in service quality that directly impact their end user experience. SLO-based alerting also ensures that they don't suffer from noisy alerts. Alerts are only triggered when the value of the SLI is such that the team is in danger of missing their SLO. ![Latency SLO dashboard](/media/docs/tempo/intro/traces-metrics-gen-SLO.png) \ No newline at end of file diff --git a/docs/sources/tempo/introduction/solutions-with-traces/traces-diagnose-errors.md b/docs/sources/tempo/introduction/solutions-with-traces/traces-diagnose-errors.md index ebfaa2c317f..01778459554 100644 --- a/docs/sources/tempo/introduction/solutions-with-traces/traces-diagnose-errors.md +++ b/docs/sources/tempo/introduction/solutions-with-traces/traces-diagnose-errors.md @@ -7,6 +7,12 @@ keywords: title: Diagnose errors with traces menuTitle: Diagnose errors with traces weight: 400 +refs: + traceql: + - pattern: /docs/tempo/ + destination: https://grafana.com/docs/tempo//traceql/ + - pattern: /docs/enterprise-traces/ + destination: https://grafana.com/docs/enterprise-traces//traceql/ --- # Diagnose errors with traces @@ -27,7 +33,7 @@ It’s imperative for the operations team at Handy Site to quickly troubleshoot ## Use TraceQL to query data -Tempo has a traces-first query language, [TraceQL](https://grafana.com/docs/tempo/latest/traceql/), that provides a unique toolset for selecting and searching tracing data. TraceQL can match traces based on span and resource attributes, duration, and ancestor<>descendant relationships. It also can compute aggregate statistics (e.g., `rate`) over a set of spans. +Tempo has a traces-first query language, [TraceQL](ref:traceql), that provides a unique toolset for selecting and searching tracing data. TraceQL can match traces based on span and resource attributes, duration, and ancestor<>descendant relationships. It also can compute aggregate statistics (e.g., `rate`) over a set of spans. Handy Site’s services and applications are instrumented for tracing, so they can use TraceQL as a debugging tool. Using three TraceQL queries, the team identifies and validates the root cause of the issue. @@ -50,7 +56,7 @@ Looking at the set of returned spans, the most concerning ones are those with th The team decides to use structural operators to follow an error chain from the top-level `mythical-requester` service to any descendant spans that also have an error status. Descendant spans can be any span that's descended from the parent span, such as a child or a further child at any depth. -Using this query, the team can pinpoint the downstream service that might be causing the issue. The query below says "Find me spans where `status = error` that that are descendants of spans from the `mythical-requester` service that have status code `500`." +Using this query, the team can pinpoint the downstream service that might be causing the issue. The query below says "Find me spans where `status = error` that that are descendants of spans from the `mythical-requester` service that have status code `500`." ```traceql { resource.service.name = "mythical-requester" && span.http.status_code = 500 } >> { status = error } @@ -68,7 +74,7 @@ Specifically, the service is passing a `null` value for a column in a database t After identifying the specific cause of this internal server error, the team wants to know if there are errors in any database operations other than the `null` `INSERT` error found above. Their updated query uses a negated regular expression to find any spans where the database statement either doesn’t exist, or doesn’t start with an `INSERT` clause. -This should expose any other issues causing an internal server error and filter out the class of issues that they already diagnosed. +This should expose any other issues causing an internal server error and filter out the class of issues that they already diagnosed. ```traceql { resource.service.name = "mythical-requester" && span.http.status_code = 500 } >> { status = error && span.db.statement !~ "INSERT.*" } @@ -76,6 +82,6 @@ This should expose any other issues causing an internal server error and filter This query yields no results, suggesting that the root cause of the issues the operations team are seeing is exclusively due to the failing database `INSERT` statement. At this point, they can roll back to a known working version of the service, or deploy a fix to ensure that `null` data being passed to the service is rejected appropriately. -Once that is complete, the issue can be marked resolved and the Handy team's error rate SLI should return back to acceptable levels. +Once that is complete, the issue can be marked resolved and the Handy team's error rate SLI should return back to acceptable levels. ![Empty query results](/media/docs/tempo/intro/traceql-no-results-handy-site.png) diff --git a/docs/sources/tempo/metrics-generator/active-series.md b/docs/sources/tempo/metrics-generator/active-series.md index 9dab55cfeb9..ba3991d2d51 100644 --- a/docs/sources/tempo/metrics-generator/active-series.md +++ b/docs/sources/tempo/metrics-generator/active-series.md @@ -16,9 +16,9 @@ These capabilities rely on a set of generated span metrics and service metrics. Any spans that are ingested by Tempo can create many metric series. However, this doesn't mean that every time a span is ingested that a new active series is created. -The number of active series generated depends on the label pairs generated from span data that are associated with the metrics, similar to other Prometheus-formated data. +The number of active series generated depends on the label pairs generated from span data that are associated with the metrics, similar to other Prometheus-formatted data. -For additional information, refer to the [Active series and DPM documentation](/docs/grafana-cloud/billing-and-usage/active-series-and-dpm/#active-series). +For additional information, refer to the [Active series and DPM documentation](https://grafana.com/docs/grafana-cloud/billing-and-usage/active-series-and-dpm/). ## Active series calculation diff --git a/docs/sources/tempo/metrics-generator/service-graph-view.md b/docs/sources/tempo/metrics-generator/service-graph-view.md index 25babf08ff2..ec53c1c7de1 100644 --- a/docs/sources/tempo/metrics-generator/service-graph-view.md +++ b/docs/sources/tempo/metrics-generator/service-graph-view.md @@ -5,6 +5,17 @@ description: Grafana's service graph view utilizes metrics generated by the metr aliases: - ./app-performance-mgmt # /docs/tempo//metrics-generator/app-performance-mgmt weight: 400 +refs: + enable-service-graphs: + - pattern: /docs/tempo/ + destination: https://grafana.com/docs/tempo//metrics-generator/service_graphs/enable-service-graphs/ + - pattern: /docs/enterprise-traces/ + destination: https://grafana.com/docs/enterprise-traces//metrics-generator/service_graphs/enable-service-graphs/ + span-metrics: + - pattern: /docs/tempo/ + destination: https://grafana.com/docs/tempo//metrics-generator/span_metrics/ + - pattern: /docs/enterprise-traces/ + destination: https://grafana.com/docs/enterprise-traces//metrics-generator/span_metrics/ --- # Service graph view @@ -27,13 +38,13 @@ You have to enable span metrics and service graph generation on the Grafana back To use the service graph view, you need: -* Tempo or Grafana Cloud Traces with either 1) the metrics generator enabled and configured or 2) Grafana Agent or Grafana Alloy enabled and configured to send data to a Prometheus-compatible metrics store -* [Services graphs]({{< relref "../metrics-generator/service_graphs/enable-service-graphs" >}}), which are enabled by default in Grafana -* [Span metrics]({{< relref "../metrics-generator/span_metrics#how-to-run" >}}) enabled in your Tempo data source configuration +* Tempo or Grafana Cloud Traces with either the metrics generator enabled and configured or Grafana Agent or Grafana Alloy enabled and configured to send data to a Prometheus-compatible metrics store +* [Services graphs](ref:enable-service-graphs), which are enabled by default in Grafana +* [Span metrics](ref:span-metrics) enabled in your Tempo data source configuration The service graph view can be derived from metrics generated by either the metrics-generator or by Grafana Agent or Grafana Alloy. -For information on how to configure these features, refer to the [Grafana Tempo data sources documentation](/docs/grafana/latest/datasources/tempo/). +For information on how to configure these features, refer to the [Tempo data sources documentation](/docs/grafana//datasources/tempo/). ## What does the service graph view show? @@ -46,7 +57,7 @@ The service graph view provides a span metrics visualization (table) and service You can select any information in the table that has an underline to show more detailed information. You can also select any node in the service graph to display additional information. -![Service graph with extended informaiton](/media/docs/grafana/data-sources/tempo/query-editor/tempo-ds-query-service-graph-prom.png) +![Service graph with extended information](/media/docs/grafana/data-sources/tempo/query-editor/tempo-ds-query-service-graph-prom.png) ### Error rate example diff --git a/docs/sources/tempo/metrics-generator/service_graphs/enable-service-graphs.md b/docs/sources/tempo/metrics-generator/service_graphs/enable-service-graphs.md index 5a01c425d9b..079a0e4246e 100644 --- a/docs/sources/tempo/metrics-generator/service_graphs/enable-service-graphs.md +++ b/docs/sources/tempo/metrics-generator/service_graphs/enable-service-graphs.md @@ -5,32 +5,38 @@ aliases: title: Enable service graphs description: Learn how to enable service graphs weight: 200 +refs: + cardinality: + - pattern: /docs/tempo/ + destination: https://grafana.com/docs/tempo//metrics-generator/cardinality/ + - pattern: /docs/enterprise-traces/ + destination: https://grafana.com/docs/enterprise-traces//metrics-generator/cardinality/ --- ## Enable service graphs Service graphs are generated in Tempo and pushed to a metrics storage. Then, they can be represented in Grafana as a graph. -You will need those components to fully use service graphs. +You need those components to fully use service graphs. {{< admonition type="note" >}} Cardinality can pose a problem when you have lots of services. -To learn more about cardinality and how to perform a dry run of the metrics generator, see the [Cardinality documentation]({{< relref "../cardinality" >}}). -{{% /admonition %}} +To learn more about cardinality and how to perform a dry run of the metrics-generator, refer to the [Cardinality documentation](ref:cardinality). +{{< /admonition >}} ### Enable service graphs in Tempo/GET To enable service graphs in Tempo/GET, enable the metrics generator and add an overrides section which enables the `service-graphs` generator. -For more information, refer to the [configuration details]({{< relref "../../configuration#metrics-generator" >}}). +For more information, refer to the [configuration details](https://grafana.com/docs/tempo//configuration#metrics-generator). -To enable service graphs when using Grafana Agent, refer to the [Grafana Agent and service graphs documentation]({{< relref "../../configuration/grafana-agent/service-graphs" >}}). +To enable service graphs when using Grafana Alloy, refer to the [Grafana Alloy and service graphs documentation](https://grafana.com/docs/tempo//configuration/grafana-alloy/service-graphs/). ### Enable service graphs in Grafana {{< admonition type="note" >}} -Since Grafana 9.0.4, service graphs have been enabled by default. Prior to Grafana 9.0.4, service graphs were hidden -under the [feature toggle](/docs/grafana/latest/setup-grafana/configure-grafana/#feature_toggles) `tempoServiceGraph`. -{{% /admonition %}} +Service graphs are enabled by default in Grafana. Prior to Grafana 9.0.4, service graphs were hidden +under the [feature toggle](/docs/grafana/latest/setup-grafana/configure-grafana) `tempoServiceGraph`. +{{< /admonition >}} Configure a Tempo data source's service graphs by linking to the Prometheus backend where metrics are being sent: diff --git a/docs/sources/tempo/metrics-generator/span_metrics.md b/docs/sources/tempo/metrics-generator/span_metrics.md index 68430e3feca..88aba4a01b5 100644 --- a/docs/sources/tempo/metrics-generator/span_metrics.md +++ b/docs/sources/tempo/metrics-generator/span_metrics.md @@ -4,6 +4,12 @@ aliases: title: Span metrics description: The span metrics processor generates metrics from ingested tracing data, including request, error, and duration (RED) metrics. weight: 200 +refs: + cardinality: + - pattern: /docs/tempo/ + destination: https://grafana.com/docs/tempo//metrics-generator/cardinality/ + - pattern: /docs/enterprise-traces/ + destination: https://grafana.com/docs/enterprise-traces//metrics-generator/cardinality/ --- # Span metrics @@ -23,7 +29,7 @@ Even if you already have metrics, span metrics can provide in-depth monitoring o The generated metrics will show application level insight into your monitoring, as far as tracing gets propagated through your applications. -Last but not least, span metrics lower the entry barrier for using [exemplars](/docs/grafana/latest/basics/exemplars/). +Last but not least, span metrics lower the entry barrier for using [exemplars](https://grafana.com/docs/grafana//basics/exemplars/). An exemplar is a specific trace representative of measurement taken in a given time interval. Since traces and metrics co-exist in the metrics-generator, exemplars can be automatically added, providing additional value to these metrics. @@ -31,7 +37,7 @@ exemplars can be automatically added, providing additional value to these metric ## How to run To enable span metrics in Tempo or Grafana Enterprise Traces, enable the metrics generator and add an overrides section which enables the `span-metrics` processor. -Refer to [the configuration details]({{< relref "../configuration#metrics-generator" >}}). +Refer to [the configuration details](https://grafana.com/docs/tempo//configuration#metrics-generator). If you want to enable metrics-generator for your Grafana Cloud account, refer to the [Metrics-generator in Grafana Cloud](https://grafana.com/docs/grafana-cloud/send-data/traces/metrics-generator/) documentation. @@ -44,8 +50,8 @@ This processor mirrored the implementation from the OpenTelemetry Collector of t The OTel `spanmetricsprocessor` has since been [deprecated](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/processor/spanmetricsprocessor/v0.95.0/processor/spanmetricsprocessor/README.md) and replaced with the [span metric connector](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/processor/spanmetricsprocessor/v0.95.0/connector/spanmetricsconnector/README.md). {{< admonition type="note" >}} -To learn more about cardinality and how to perform a dry run of the metrics generator, see the [Cardinality documentation]({{< relref "./cardinality" >}}). -{{% /admonition %}} +To learn more about cardinality and how to perform a dry run of the metrics generator, see the [Cardinality documentation](ref:cardinality). +{{< /admonition >}} ### Metrics @@ -59,32 +65,32 @@ The following metrics are exported: {{< admonition type="note" >}} In Tempo 1.4 and 1.4.1, the histogram metric was called `traces_spanmetrics_duration_seconds`. This was changed later to be consistent with the metrics generated by Grafana Agent and the OpenTelemetry Collector. -{{% /admonition %}} +{{< /admonition >}} By default, the metrics processor adds the following labels to each metric: `service`, `span_name`, `span_kind`, `status_code`, `status_message`, `job`, and `instance`. - `service` - The name of the service that generated the span - `span_name` - The unique name of the span - `span_kind` - The type of span, this can be one of five values: - - `SPAN_KIND_SERVER` - The span was generated by a call from another service - - `SPAN_KIND_CLIENT` - The span made a call to another service - - `SPAN_KIND_INTERNAL` - The span does not have interaction outside of the service it was generated in - - `SPAN_KIND_PUBLISHER` - The span created data that was pushed onto a bus or message broker - - `SPAN_KIND_CONSUMER` - The span consumed data that was on a bus or messaging system + - `SPAN_KIND_SERVER` - The span was generated by a call from another service + - `SPAN_KIND_CLIENT` - The span made a call to another service + - `SPAN_KIND_INTERNAL` - The span does not have interaction outside of the service it was generated in + - `SPAN_KIND_PUBLISHER` - The span created data that was pushed onto a bus or message broker + - `SPAN_KIND_CONSUMER` - The span consumed data that was on a bus or messaging system - `status_code` - The result of the span, this can be one of three values: - - `STATUS_CODE_UNSET` - Result of the span was unset/unknown - - `STATUS_CODE_OK` - The span operation completed successfully - - `STATUS_CODE_ERROR` - The span operation completed with an error + - `STATUS_CODE_UNSET` - Result of the span was unset/unknown + - `STATUS_CODE_OK` - The span operation completed successfully + - `STATUS_CODE_ERROR` - The span operation completed with an error - `status_message` (optionally enabled) - The message that details the reason for the `status_code` label - `job` - The name of the job, a combination of namespace and service; only added if `metrics_generator.processor.span_metrics.enable_target_info: true` - `instance` - The instance ID; only added if `metrics_generator.processor.span_metrics.enable_target_info: true` -Additional user defined labels can be created using the [`dimensions` configuration option]({{< relref "../configuration#metrics-generator" >}}). -When a configured dimension collides with one of the default labels (e.g. `status_code`), the label for the respective dimension is prefixed with double underscore (i.e. `__status_code`). +Additional user defined labels can be created using the [`dimensions` configuration option](https://grafana.com/docs/tempo//configuration#metrics-generator). +When a configured dimension collides with one of the default labels (for example, `status_code`), the label for the respective dimension is prefixed with double underscore (for example, `__status_code`). -Custom labeling of dimensions is also supported using the [`dimension_mapping` configuration option]({{< relref "../configuration#metrics-generator" >}}). +Custom labeling of dimensions is also supported using the [`dimension_mapping` configuration option](https://grafana.com/docs/tempo//configuration#metrics-generator). -An optional metric called `traces_target_info` using all resource level attributes as dimensions can be enabled in the [`enable_target_info` configuration option]({{< relref "../configuration#metrics-generator" >}}). +An optional metric called `traces_target_info` using all resource level attributes as dimensions can be enabled in the [`enable_target_info` configuration option](https://grafana.com/docs/tmepo//configuration#metrics-generator). If you use a ratio-based sampler, you can use the custom sampler below to not lose metric information. However, you also need to set `metrics_generator.processor.span_metrics.span_multiplier_key` to `"X-SampleRatio"`. diff --git a/docs/sources/tempo/traceql/metrics-queries/_index.md b/docs/sources/tempo/traceql/metrics-queries/_index.md index 817fd5dd005..890634a0480 100644 --- a/docs/sources/tempo/traceql/metrics-queries/_index.md +++ b/docs/sources/tempo/traceql/metrics-queries/_index.md @@ -6,6 +6,17 @@ weight: 600 keywords: - metrics query - TraceQL metrics +refs: + solve-problems-mq: + - pattern: /docs/tempo/ + destination: https://grafana.com/docs/tempo//traceql/metrics-queries/solve-problems-metrics-queries/ + - pattern: /docs/enterprise-traces/ + destination: https://grafana.com/docs/enterprise-traces//traceql/metrics-queries/solve-problems-metrics-queries/ + mq-functions: + - pattern: /docs/tempo/ + destination: https://grafana.com/docs/tempo//traceql/metrics-queries/functions/ + - pattern: /docs/enterprise-traces/ + destination: https://grafana.com/docs/enterprise-traces//traceql/metrics-queries/functions/ --- # TraceQL metrics queries @@ -43,7 +54,7 @@ For more information about the RED method, refer to [The RED Method: how to inst You can write TraceQL metrics queries to compute rate, errors, and durations over different groups of spans. -For more information on how to use TraceQL metrics to investigate issues, refer to [Solve problems with metrics queries](./solve-problems-metrics-queries). +For more information on how to use TraceQL metrics to investigate issues, refer to [Solve problems with metrics queries](ref:solve-problems-mq). ## Enable and use TraceQL metrics @@ -56,14 +67,14 @@ To run TraceQL metrics queries in Grafana, you need Grafana Cloud or Grafana 10. No extra configuration is needed. Use a Tempo data source that points to a Tempo database with TraceQL metrics enabled. -Refer to [Solve problems using metrics queries](./solve-problems-metrics-queries/) for some real-world examples. +Refer to [Solve problems using metrics queries](ref:solve-problems-mq) for some real-world examples. ### Functions TraceQL metrics queries currently include the following functions for aggregating over groups of spans: `rate`, `count_over_time`, `max_over_time`, `min_over_time`, `avg_over_time`, `quantile_over_time`, `histogram_over_time`, and `compare`. These functions can be added as an operator at the end of any TraceQL query. -For detailed information and example queries for each function, refer to [TraceQL metrics functions](./functions). +For detailed information and example queries for each function, refer to [TraceQL metrics functions](ref:mq-functions). ### Exemplars diff --git a/docs/sources/tempo/traceql/metrics-queries/functions.md b/docs/sources/tempo/traceql/metrics-queries/functions.md index 3116946b22a..16ab4307d8f 100644 --- a/docs/sources/tempo/traceql/metrics-queries/functions.md +++ b/docs/sources/tempo/traceql/metrics-queries/functions.md @@ -22,7 +22,7 @@ These functions can be added as an operator at the end of any TraceQL query. : Calculates the number of matching spans per second `count_over_time` -: Counts the number of matching spans per time interval (refer to the [`step` API parameter](https://grafana.com/docs/tempo//api_docs/#traceql-metrics)). +: Counts the number of matching spans per time interval (refer to the [`step` API parameter](https://grafana.com/docs/tempo//api_docs)). `min_over_time` : Returns the minimum value for the specified attribute across all matching spans per time interval (refer to the [`step` API parameter](https://grafana.com/docs/tempo//api_docs/#traceql-metrics)). @@ -87,7 +87,6 @@ The `count_over_time()` function counts the number of matching spans per time in The time interval that the count will be computed over is set by the `step` parameter. For more information, refer to the [`step` API parameter](https://grafana.com/docs/tempo//api_docs/#traceql-metrics). - ### Example This example counts the number of spans with name `"GET /:endpoint"` broken down by status code. You might see that there are 10 `"GET /:endpoint"` spans with status code 200 and 15 `"GET /:endpoint"` spans with status code 400. @@ -186,7 +185,7 @@ You could use a similar query to know what the 50th percentile and 95th percenti The `compare` function is used to split a set of spans into two groups: a selection and a baseline. It returns time-series for all attributes found on the spans to highlight the differences between the two groups. -This is a powerful function that's best understood by using the [**Comparison** tab in Explore Traces](https://grafana.com/docs/grafana//explore/simplified-exploration/traces/investigate/#comparison). +This is a powerful function that's best understood by using the [**Comparison** tab in Explore Traces](https://grafana.com/docs/grafana//explore/simplified-exploration/traces/investigate/#comparison). You can also under this function by looking at example outputs below. The function is used like other metrics functions: when it's placed after any trace query, it converts the query into a metrics query: diff --git a/docs/sources/tempo/traceql/metrics-queries/solve-problems-metrics-queries.md b/docs/sources/tempo/traceql/metrics-queries/solve-problems-metrics-queries.md index 7f2c65730c8..aa2bfb55464 100644 --- a/docs/sources/tempo/traceql/metrics-queries/solve-problems-metrics-queries.md +++ b/docs/sources/tempo/traceql/metrics-queries/solve-problems-metrics-queries.md @@ -6,6 +6,12 @@ weight: 600 keywords: - metrics query - TraceQL metrics +refs: + metrics-generator: + - pattern: /docs/tempo/ + destination: https://grafana.com/docs/tempo//metrics-generator/ + - pattern: /docs/enterprise-traces/ + destination: https://grafana.com/docs/enterprise-traces//metrics-generator/ --- # Solve problems with trace metrics queries @@ -18,8 +24,9 @@ This page provides an example of how you can investigate the rate of incoming re ## RED metrics and queries The Tempo metrics-generator emits metrics with pre-configured labels for Rate, Error, and Duration (RED) metrics and service graph edges. -Generated metric labels vary, but always include the service name (in service graph metrics, as a client and/or a server type). -For more information, refer to the [metrics-generator documentation](../../metrics-generator/). +Generated metric labels vary, but always include the service name. +For example, in service graph metrics, as a client or a server type, or both. +For more information, refer to the [metrics-generator documentation](ref:metrics-generator). You can use these metrics to get an overview of application performance. The metrics can be directly correlated to the trace spans that are available for querying. @@ -27,7 +34,8 @@ The metrics can be directly correlated to the trace spans that are available for TraceQL metrics allow a user to query metrics from traces directly from Tempo instead of requiring the metrics-generator component and an accompanying TSDB. {{< admonition type="note" >}} -TraceQL metrics are constrained to a 24-hour range window, and aren't available as a Grafana Managed Alerts source. For any metrics that you want to query over longer time ranges, use for alerting, or retain for more than 30 days, use the metrics-generator to store these metrics in Prometheus, Mimir, or other Prometheus-compatible TSDB and continue to use PromQL for querying. +TraceQL metrics are constrained to a 24-hour range window, and aren't available as a Grafana Managed Alerts source. +For any metrics that you want to query over longer time ranges, use for alerting, or retain for more than 30 days, use the metrics-generator to store these metrics in Prometheus, Mimir, or other Prometheus-compatible TSDB and continue to use PromQL for querying. {{< /admonition >}} ## Investigate the rate of incoming requests diff --git a/docs/sources/tempo/traceql/query-editor/_index.md b/docs/sources/tempo/traceql/query-editor/_index.md index 858afaa7d3a..c22163d9647 100644 --- a/docs/sources/tempo/traceql/query-editor/_index.md +++ b/docs/sources/tempo/traceql/query-editor/_index.md @@ -11,14 +11,25 @@ keywords: - Tempo query language - query editor - TraceQL +refs: + traceql: + - pattern: /docs/tempo/ + destination: https://grafana.com/docs/tempo//traceql/ + - pattern: /docs/enterprise-traces/ + destination: https://grafana.com/docs/enterprise-traces//traceql/ + service-graph-view: + - pattern: /docs/tempo/ + destination: https://grafana.com/docs/tempo//metrics-generator/service-graph-view/ + - pattern: /docs/enterprise-traces/ + destination: https://grafana.com/docs/enterprise-traces//metrics-generator/service-graph-view/ --- # Write TraceQL queries in Grafana The Tempo data source's query editor helps you query and display traces from Tempo in [Explore](https://grafana.com/docs/grafana//explore/). -The queries use [TraceQL](/docs/tempo/latest/traceql), the query language designed specifically for tracing. +The queries use [TraceQL](ref:traceql), the query language designed specifically for tracing. -For general documentation on querying data sources in Grafana, refer to [Query and transform data](/docs/grafana//panels-visualizations/query-transform-data/). +For general documentation on querying data sources in Grafana, refer to [Query and transform data](https://grafana.com/docs/grafana//panels-visualizations/query-transform-data/). ## Before you begin @@ -35,7 +46,7 @@ The three **Query types** are: - The **Search** query builder provides a user interface for building a TraceQL query. - The **TraceQL** query editor lets you write your own TraceQL query with assistance from autocomplete. -- The **Service Graph** view displays a visual relationship between services. Refer to the [Service graph view](https://grafana.com/docs/tempo//metrics-generator/service-graph-view/) documentation for more information. +- The **Service Graph** view displays a visual relationship between services. Refer to the [Service graph view](ref:service-graph-view) documentation for more information. ### Search query builder @@ -68,7 +79,7 @@ Using the service graph view, you can: - Determine how long the slowest queries in your service take to complete - Examine all traces that contain spans of particular interest based on rate, error, and duration values (RED signals) -For more information about the service graph, refer to [Service graph view](https://grafana.com/docs/tempo//metrics-generator/service-graph-view/). +For more information about the service graph, refer to [Service graph view](ref:service-graph-view). ![Screenshot of the Service Graph view](/media/docs/grafana/data-sources/tempo/query-editor/tempo-ds-query-service-graph.png)