Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add doc about profiling Alloy #1781

Merged
merged 3 commits into from
Oct 7, 2024
Merged
Changes from 1 commit
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
101 changes: 101 additions & 0 deletions docs/sources/troubleshoot/profile.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,101 @@
---
canonical: https://grafana.com/docs/alloy/latest/troubleshoot/profile/
description: Learn how to profile resource consumption
title: Profile Grafana Alloy resource consumption
menuTitle: Profile resource consumption
weight: 300
---

# Profile Grafana Alloy resource consumption
ptodev marked this conversation as resolved.
Show resolved Hide resolved

Alloy is written in the Go programming language, which offers [built-in][pprof-pkg] support for profiling.
ptodev marked this conversation as resolved.
Show resolved Hide resolved
Just like other applications written in Go, you can profile Alloy by sending an HTTP request which returns you a "pprof" Go profile.
ptodev marked this conversation as resolved.
Show resolved Hide resolved

Once you have the pprof file, simply visualize it as a [flame graph][flame-graph] in [Grafana Pyroscope][pyroscope-getstarted].
Alternatively, you could visualize it on [Grafana Play][pyroscope-adhoc] or by using Go's built-in [pprof tool][go-pprof] locally.
ptodev marked this conversation as resolved.
Show resolved Hide resolved

{{< admonition type="note" >}}
A profile may contain sensitive information about your environment.
You may not want to upload your profiles to a public location.
{{< /admonition >}}

The port on which you need to send the HTTP request is governed by Alloy's `--server.http.listen-addr` [command line argument][cmd-cli].
ptodev marked this conversation as resolved.
Show resolved Hide resolved
It is set to `127.0.0.1:12345` by default.

[pprof-pkg]: https://pkg.go.dev/net/http/pprof
[pyroscope-adhoc]: https://play.grafana.org/a/grafana-pyroscope-app/ad-hoc
[go-pprof]: https://go.dev/blog/pprof
[pyroscope-getstarted]: https://grafana.com/docs/pyroscope/latest/get-started/
[flame-graph]: https://grafana.com/docs/pyroscope/latest/view-and-analyze-profile-data/flamegraphs/
[cmd-cli]: ../../reference/cli/run

## Obtaining a single profile

Different types of HTTP requests will retrieve different profiles.

### Memory consumption

Memory leaks are often caused by goroutine leaks.
ptodev marked this conversation as resolved.
Show resolved Hide resolved
This is why obtaining a goroutine profile is usually necessary when investigating memory issues.
For example:

```
curl localhost:12345/debug/pprof/heap -o heap.pprof
curl localhost:12345/debug/pprof/goroutine -o goroutine.pprof
```
ptodev marked this conversation as resolved.
Show resolved Hide resolved

It is often helpful to collect profiles both when the memory usage is low, and also when it is high.
ptodev marked this conversation as resolved.
Show resolved Hide resolved
That way it may be more clear what caused the consumption to increase.
ptodev marked this conversation as resolved.
Show resolved Hide resolved

### CPU consumption

If you are experiencing high CPU consumption, you can collect a CPU profile:

```
curl http://localhost:12345/debug/pprof/profile?seconds=30 -o cpu.pprof
```
ptodev marked this conversation as resolved.
Show resolved Hide resolved

The `?seconds=30` part of the URL above means that the profiling will continue for 30 seconds.
ptodev marked this conversation as resolved.
Show resolved Hide resolved

## Continuous profiling

You don't have to send manual `curl` commands every time you want to collect profiles.
You can also profile continuously using Alloy's built-in [pyroscope components][].
ptodev marked this conversation as resolved.
Show resolved Hide resolved

If you have very few Alloy instances, you can even configure them to profile themselves.
However, if you have a large cluster of collectors, it is best to set up Alloy instances whose sole job is to profile other Alloy instances.
ptodev marked this conversation as resolved.
Show resolved Hide resolved

An example of an Alloy instance profiling itself:
ptodev marked this conversation as resolved.
Show resolved Hide resolved

```alloy
pyroscope.scrape "default" {
targets = [{"__address__" = "localhost:12345", "service_name"="alloy"}]
forward_to = [pyroscope.write.default.receiver]
}

pyroscope.write "default" {
endpoint {
url = "https://profiles-prod-014.grafana.net"
basic_auth {
username = sys.env("PYROSCOPE_USERNAME")
password = sys.env("PYROSCOPE_PASSWORD")
}
}
}
```

[pyroscope components]: ../../reference/components/pyroscope

## What is Alloy's expected resource consumption?
ptodev marked this conversation as resolved.
Show resolved Hide resolved

Refer to the [Estimate resource usage][res-usage] page.
ptodev marked this conversation as resolved.
Show resolved Hide resolved

[res-usage]: ../../introduction/estimate-resource-usage

## What if my Alloy instance consumes an abnormally large amount of resources?
ptodev marked this conversation as resolved.
Show resolved Hide resolved

You could open an issue in [Alloy's repository][alloy-repo].
It would be helpful to attach pprof files and Alloy's configuration file.
Make sure any secrets are redacted.
ptodev marked this conversation as resolved.
Show resolved Hide resolved

[alloy-repo]: https://github.com/grafana/alloy/issues
Loading