Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[DOC] Update metrics for Alloy troubleshooting doc #4137

Merged
Show file tree
Hide file tree
Changes from 4 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
18 changes: 9 additions & 9 deletions docs/sources/tempo/troubleshooting/_index.md
Original file line number Diff line number Diff line change
Expand Up @@ -16,18 +16,18 @@ In addition, the [Tempo runbook](https://github.com/grafana/tempo/blob/main/oper

## Sending traces

- [Spans are being refused with "pusher failed to consume trace data"]({{< relref "./max-trace-limit-reached" >}})
- [Is Grafana Alloy sending to the backend?]({{< relref "./agent" >}})
- [Spans are being refused with "pusher failed to consume trace data"](https://grafana.com/docs/tempo/<TEMMPO_VERSION>/troubleshooting/max-trace-limit-reached/)
- [Is Grafana Alloy sending to the backend?](https://grafana.com/docs/tempo/<TEMPO_VERSION>/troubleshooting/alloy/)

## Querying

- [Unable to find my traces in Tempo]({{< relref "./unable-to-see-trace" >}})
- [Error message "Too many jobs in the queue"]({{< relref "./too-many-jobs-in-queue" >}})
- [Queries fail with 500 and "error using pageFinder"]({{< relref "./bad-blocks" >}})
- [I can search traces, but there are no service name or span name values available]({{< relref "./search-tag" >}})
- [Error message `response larger than the max (<number> vs <limit>)`]({{< relref "./response-too-large" >}})
- [Search results don't match trace lookup results with long-running traces]({{< relref "./long-running-traces" >}})
- [Unable to find my traces in Tempo](https://grafana.com/docs/tempo/<TEMPO_VERSION>/troubleshooting/unable-to-see-trace/)
- [Error message "Too many jobs in the queue"](https://grafana.com/docs/tempo/<TEMPO_VERSION>/troubleshooting/too-many-jobs-in-queue/)
- [Queries fail with 500 and "error using pageFinder"](https://grafana.com/docs/tempo/<TEMPO_VERSION>/troubleshooting/bad-blocks/)
- [I can search traces, but there are no service name or span name values available](https://grafana.com/docs/tempo/<TEMPO_VERSION>/troubleshooting/search-tag)
- [Error message `response larger than the max (<number> vs <limit>)`](https://grafana.com/docs/tempo/<TEMPO_VERSION>/troubleshooting/response-too-large/)
- [Search results don't match trace lookup results with long-running traces](https://grafana.com/docs/tempo/<TEMPO_VERSION>/troubleshooting/long-running-traces/)

## Metrics-generator

- [Metrics or service graphs seem incomplete]({{< relref "./metrics-generator" >}})
- [Metrics or service graphs seem incomplete](https://grafana.com/docs/tempo/<TEMPO_VERSION>/troubleshooting/metrics-generator/)
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,7 @@ description: Gain visibility on how many traces are being pushed to Grafana Allo
weight: 472
aliases:
- ../operations/troubleshooting/agent/
- ./agent.md # /docs/tempo/<TEMPO_VERSION>/troubleshooting/agent.md
---

# Troubleshoot Grafana Alloy
Expand Down Expand Up @@ -33,6 +34,24 @@ exporter_sent_spans_ratio_total
exporter_send_failed_spans_ratio_total
```

If you are using Grafana Alloy to send traces to Grafana Cloud, the metrics are visible at
`http://localhost:12345/metrics`.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a bit misleading, this endpoint is local and it states Grafana Cloud

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is referring to the metrics emitted by Alloy itself, /metrics is its prometheus scrape endpoint. These should also end up in Grafana Cloud, but in case remote write isn't working yet this is a quick and easy way to check the metrics locally.

I think the bit that might be confusing is If you are using Grafana Alloy since this page is about Alloy already.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good points. I"ll write a suggestion to clarifiy.

knylander-grafana marked this conversation as resolved.
Show resolved Hide resolved
The `/metrics` HTTP endpoint of the Alloy HTTP server exposes the Alloy component and controller metrics.
Refer to the [Monitor the Grafana Alloy component controller](https://grafana.com/docs/alloy/latest/troubleshoot/controller_metrics/) documentation for more information.

### Check metrics in Grafana Cloud

In your Grafana Cloud instance, you can check metrics using the `grafanacloud-usage` data source.
To view the metrics, use the following steps:

1. From your Grafana instance, select **Explore** in the left menu.
1. Change the data source to `grafanacloud-usage`.
1. Type the metric to verify in the text box. If you start with `grafanacloud_traces_`, you can use autocomplete to browse the list of available metrics.

Refer to [Cloud Traces usage metrics](https://grafana.com/docs/grafana-cloud/cost-management-and-billing/understand-your-invoice/usage-limits/#cloud-traces-usage) for a list of metrics related to tracing usage.

![Use Explore to check the metrics for traces sent to Grafana Cloud](/media/docs/tempo/screenshot-tempo-trouble-metrics-search.png)

## Trace span logging

If metrics and logs are looking good, but you are still unable to find traces in Grafana Cloud, you can configure Alloy to output all the traces it receives to the [console](https://grafana.com/docs/tempo/<TEMPO_VERSION>/configuration/grafana-alloy/automatic-logging/).