Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

First draft of a doc for Tracing in networking #42830

Draft
wants to merge 3 commits into
base: main
Choose a base branch
from
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
122 changes: 24 additions & 98 deletions docs/fundamentals/networking/telemetry/metrics.md
Original file line number Diff line number Diff line change
Expand Up @@ -26,122 +26,48 @@

This section demonstrates various methods to collect and view System.Net metrics.

### Example app
### .NET Aspire
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Are we sure we want to entirely replace this paragraph instead of eg. only replacing the prometheus+grafana part with the aspire example? The original example app is simple and self-contained and the paragraph also demonstrates dotnet-monitor.


For the sake of this tutorial, create a simple app that sends HTTP requests to various endpoints in parallel.
The simplest solution for collecting metrics for ASP.NET applications is to use [.NET Aspire](/dotnet/aspire/get-started/aspire-overview) which is a set of extensions to .NET to make it easy to create and work with distributed applications. One of the benefits of using .NET Aspire is that telemetry is built in, using the OpenTelemetry libraries for .NET. The default project templates for .NET Aspire contain a `ServiceDefaults` project, part of which is to setup and configure OTel. The Service Defaults project is referenced and initialized by each service in a .NET Aspire solution.
Copy link
Member

@antonfirsov antonfirsov Oct 9, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

for ASP.NET applications

This article is not supposed to be about ASP.NET, it should focus on System.Net.* stuff.

The simplest solution

I would prefer to use a less opinionated language. In our docs we now reference a wide range of tools (Application Insights, Prometheus+Grafana and now Aspire) which became somewhat confusing on it's own. Assuming that we don't plan to harmonize everything (=push for a particular solution against others) I would let the user decide what is simplest for their use-case.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This telemetry is mostly going to be used in server apps - which today means ASP.NET. Its less about the web server and more about the host + builder pattern which is the required infrastructure.

What we should stop doing is having many different docs talk about the same concepts but implementing them differently. Its very confusing for users to know what patterns that they should be using. I think uniting around ServiceDefaults is going to provide the common infrastructure (that doesn't need to change on a per-project basis) and lead to more success.


```dotnetcli
dotnet new console -o HelloBuiltinMetrics
cd ..\HelloBuiltinMetrics
```

Replace the contents of `Program.cs` with the following sample code:

:::code language="csharp" source="snippets/metrics/Program.cs" id="snippet_ExampleApp":::

### View metrics with dotnet-counters

[`dotnet-counters`](../../../core/diagnostics/dotnet-counters.md) is a cross-platform performance monitoring tool for ad-hoc health monitoring and first-level performance investigation.
The Service Defaults project template includes the OTel SDK, ASP.NET, HttpClient and Runtime Instrumentation packages, and those are configured in the [`Extensions.cs`](https://github.com/dotnet/aspire/blob/main/src/Aspire.ProjectTemplates/templates/aspire-servicedefaults/Extensions.cs) file. For exporting telemetry .NET Aspire includes the OTLP exporter by default so that it can provide telemetry visualization using the Aspire Dashboard.

```dotnetcli
dotnet tool install --global dotnet-counters
```

When running against a .NET 8+ process, `dotnet-counters` enables the instruments defined by the `--counters` argument and displays the measurements. It continuously refreshes the console with the latest numbers:

```console
dotnet-counters monitor --counters System.Net.Http,System.Net.NameResolution -n HelloBuiltinMetrics
```
The Aspire Dashboard is designed to bring telemetry observation to the local debug cycle, which enables developers to not only ensure that the applications are producing telemetry, but also use that telemetry to diagnose those applications locally. Being able to observe the calls between services is proving to be just as useful at debug time as in production. The .NET Aspire dashboard is launched automatically when you F5 the `AppHost` Project from Visual Studio or `dotnet run` the `AppHost` project.

### View metrics in Grafana with OpenTelemetry and Prometheus
[![Aspire Dashboard](../../../core/diagnostics/media/aspire-dashboard-metrics-thumb.png)](../../../core/diagnostics/media/aspire-dashboard-metrics.png#lightbox)

#### Overview
For more details on .NET Aspire see:

[OpenTelemetry](https://opentelemetry.io/):
- [Aspire Overview](/dotnet/aspire/get-started/aspire-overview)
- [Telemetry in Aspire](/dotnet/aspire/fundamentals/telemetry)
- [Aspire Dashboard](/dotnet/aspire/fundamentals/dashboard/explore)

- Is a vendor-neutral, open-source project supported by the [Cloud Native Computing Foundation](https://www.cncf.io/).
- Standardizes generating and collecting telemetry for cloud-native software.
- Works with .NET using the .NET metric APIs.
- Is endorsed by [Azure Monitor](/azure/azure-monitor/app/opentelemetry-overview) and many APM vendors.
### Reusing Service Defaults project without .NET Aspire Orchestration

This tutorial shows one of the integrations available for OpenTelemetry metrics using the OSS [Prometheus](https://prometheus.io/) and [Grafana](https://grafana.com/) projects. The metrics data flow consists of the following steps:
Probably the easiest way to configure OTel for ASP.NET projects is to use the Aspire Service Defaults project, even if not using the rest of .NET Aspire such as the AppHost for orchestration. The Service Defaults project is available as a project template via Visual Studio or `dotnet new`. It configures OTel and sets up the OTLP exporter. You can then use the [OTel environment variables](https://github.com/open-telemetry/opentelemetry-dotnet/tree/main/src/OpenTelemetry.Exporter.OpenTelemetryProtocol#exporter-configuration) to configure the OTLP endpoint to send telemetry to, and provide the resource properties for the application.

1. The .NET metric APIs record measurements from the example app.
1. The OpenTelemetry library running in the app aggregates the measurements.
1. The Prometheus exporter library makes the aggregated data available via an HTTP metrics endpoint. 'Exporter' is what OpenTelemetry calls the libraries that transmit telemetry to vendor-specific backends.
1. A Prometheus server:
The steps to use *ServiceDefaults* outside .NET Aspire are:

- Polls the metrics endpoint.
- Reads the data.
- Stores the data in a database for long-term persistence. Prometheus refers to reading and storing data as *scraping* an endpoint.
- Can run on a different machine.
- Add the *ServiceDefaults* project to the solution using Add New Project in Visual Studio, or use `dotnet new aspire-servicedefaults --output ServiceDefaults`
- Reference the *ServiceDefaults* project from your ASP.NET application. In Visual Studio use "Add -> Project Reference" and select the *ServiceDefaults* project"
- Call its OpenTelemetry setup function as part of your application builder initialization.

1. The Grafana server:
``` csharp
var builder = WebApplication.CreateBuilder(args);
builder.ConfigureOpenTelemetry();

- Queries the data stored in Prometheus and displays it on a web-based monitoring dashboard.
- Can run on a different machine.
var app = builder.Build();

#### Configure the example app to use OpenTelemetry's Prometheus exporter
app.MapGet("/", () => "Hello World!");

Add a reference to the OpenTelemetry Prometheus exporter to the example app:

```dotnetcli
dotnet add package OpenTelemetry.Exporter.Prometheus.HttpListener --prerelease
app.Run();
```

> [!NOTE]
> This tutorial uses a pre-release build of OpenTelemetry's Prometheus support available at the time of writing.

Update `Program.cs` with OpenTelemetry configuration:

:::code language="csharp" source="snippets/metrics/Program.cs" id="snippet_PrometheusExporter" highlight="5-8":::

In the preceding code:

- `AddMeter("System.Net.Http", "System.Net.NameResolution")` configures OpenTelemetry to transmit all the metrics collected by the built-in `System.Net.Http` and `System.Net.NameResolution` meters.
- `AddPrometheusHttpListener` configures OpenTelemetry to expose Prometheus' metrics HTTP endpoint on port `9184`.

> [!NOTE]
> This configuration differs for ASP.NET Core apps, where metrics are exported with `OpenTelemetry.Exporter.Prometheus.AspNetCore` instead of `HttpListener`. See the [related ASP.NET Core example](/aspnet/core/log-mon/metrics/metrics#create-the-starter-app).

Run the app and leave it running so measurements can be collected:

```dotnetcli
dotnet run
```

#### Set up and configure Prometheus

Follow the [Prometheus first steps](https://prometheus.io/docs/introduction/first_steps/) to set up a Prometheus server and confirm it is working.

Modify the *prometheus.yml* configuration file so that Prometheus scrapes the metrics endpoint that the example app is exposing. Add the following highlighted text in the `scrape_configs` section:

:::code language="yaml" source="snippets/metrics/prometheus.yml" highlight="31-99":::

#### Start prometheus

1. Reload the configuration or restart the Prometheus server.
1. Confirm that OpenTelemetryTest is in the UP state in the **Status** > **Targets** page of the Prometheus web portal.
![Prometheus status](~/docs/core/diagnostics/media/prometheus-status.png)

1. On the Graph page of the Prometheus web portal, enter `http` in the expression text box and select `http_client_active_requests`.
![http_client_active_requests](~/docs/fundamentals/networking/telemetry/media/prometheus-search.png)
In the graph tab, Prometheus shows the value of the `http.client.active_requests` counter that's emitted by the example app.
![Prometheus active requests graph](~/docs/fundamentals/networking/telemetry/media/prometheus-active-requests.png)

#### Show metrics on a Grafana dashboard

1. Follow the [standard instructions](https://prometheus.io/docs/visualization/grafana/#installing) to install Grafana and connect it to a Prometheus data source.

1. Create a Grafana dashboard by selecting the **+** icon on the top toolbar then selecting **Dashboard**. In the dashboard editor that appears, enter **Open HTTP/1.1 Connections** in the **Title** box and the following query in the PromQL expression field:

```
sum by(http_connection_state) (http_client_open_connections{network_protocol_version="1.1"})
```
For a full walkthrough, see [Example: Use OpenTelemetry with OTLP and the standalone Aspire Dashboard](../../../core/diagnostics/observability-otlp-example.md).

![Grafana HTTP/1.1 Connections](~/docs/fundamentals/networking/telemetry/media/grafana-connections.png)
### Collecting metrics manually

1. Select **Apply** to save and view the new dashboard. It displays the number of active vs idle HTTP/1.1 connections in the pool.
For a walkthrough of how to collect metrics, as well as distributed traces without using Aspire Service Defaults, see [Example: Use OpenTelemetry with Prometheus, Grafana, and Jaeger](../../../core/diagnostics/observability-prgrja-example.md).

Check failure on line 70 in docs/fundamentals/networking/telemetry/metrics.md

View workflow job for this annotation

GitHub Actions / lint

Trailing spaces

docs/fundamentals/networking/telemetry/metrics.md:70:245 MD009/no-trailing-spaces Trailing spaces [Expected: 0 or 2; Actual: 1] https://github.com/DavidAnson/markdownlint/blob/v0.34.0/doc/md009.md

## Enrichment

Expand Down
Loading
Loading