Open Telemetry Integration

[wcalderipe]

This PR introduces OpenTelemetry (OTEL) instrumentation for all applications.

The OTEL integration in our stack is implemented through two separate code paths:

• The @narval/open-telemetry library handles Node.js instrumentation registration and exporter configuration.
• The open-telemetry module in @narval/nestjs-shared exposes NestJS services for traces and metrics, leveraging dependency injection for improved testability.

OTEL auto-instruments many popular packages (full list here). In addition to these default instrumentations, this PR adds Prisma instrumentation for database visibility. Note that the Prisma instrumentation could alternatively be implemented at the infrastructure level through database instrumentation.

Outgoing and Incoming HTTP Requests

For HTTP request tracing:

• Incoming requests: @opentelemetry/instrumentation-http sets trace context on active spans
• Outgoing requests: W3CTraceContextPropagator and W3CBaggagePropagator automatically set trace context headers

This enables continuous tracing across distributed processes.

Logs

Winston logs automatically include trace context when a span context exists, thanks to auto-instrumentation of @opentelemetry/instrumentation-winston.

2024-10-30T14:23:23.449Z [INFO]: GET /v1/data/entities 304
{
  "trace_id": "e29f3d185fa0e3e4ff547c8475a72baa",
  "span_id": "21ccbfd64f5d16a2",
  "trace_flags": "01",
  "timestamp": "2024-10-30T14:23:23.449Z"
}

The logs are transported to OTEL's log provider using @opentelemetry/winston-transport. The transport is auto-instrumented by the SDK when winston, @opentelemetry/instrumentation-winston and @opentelemetry/winston-transport packages are installed.

Testing this PR

Important

In development, OTEL's collector is disabled by default. If you want to enable it, see the OTEL section in the README.

Start Jaeger and OTEL's collector containers.

make docker/otel/up

Add the following to the .env of each application.

OTEL_SDK_DISABLED=true
OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4318
OTEL_LOGS_EXPORTER=otlp

Note

You can see the list of the SDK's environment variables here.

Or, alternatively, override your local .env with the new defaults.

make armory/copy-default-env
make policy-engine/copy-default-env
make vault/copy-default-env

Restart the applications and make some requests to the API.

Now, head over to Jaeger's http://localhost:16686

Debug OTEL SDK

To debug what's going on within OTEL's SDK and all its magical auto-instrumentation, set the diagnostic log level in the instrumentOpenTelemetry function in each main.ts file:

import { instrumentOpenTelemetry } from '@narval/open-telemetry'
import { DiagLogLevel } from '@opentelemetry/api'

instrumentOpenTelemetry({ 
  serviceName: 'auth',
  diagLogLevel: DiagLogLevel.DEBUG // OR DiagLogLevel.INFO
})

• DiagLogLevel.DEBUG: Logs everything verbosely - setup, instrumentation loading, exported spans and metrics
• DiagLogLevel.INFO: Shows minimal logs for setup, instrumentation loading, and errors

Missing Prisma Traces

If Prisma traces aren't appearing, regenerate the database clients:

make armory/db/generate-types
make policy-engine/db/generate-types
make vault/db/generate-types

This is required because the PR adds two preview features to the Prisma client: tracing and interactiveTransactions, which need client regeneration to take effect.

See more https://www.prisma.io/docs/orm/prisma-client/observability-and-logging/opentelemetry-tracing

Possible Improvements

Use decorators to reduce boilerplate when creating OpenTelemetry spans:

• @Counter(name, options?) - Increments a metric when method is called
• @ExecutionTime - Measures method execution time with a gauge metric
• @SpanAttribute(name) - Adds method argument as span attribute (similar to NestJS @Query)
• @Span(name, options?) - Creates span around method execution, using class name, method name and arguments in span name (e.g. FooService.baz)

Reference

• https://github.com/open-telemetry/opentelemetry-js-contrib/tree/main/metapackages/auto-instrumentations-node#readme
• https://jeremymorrell.dev/blog/a-practitioners-guide-to-wide-events/
• https://github.com/MetinSeylan/Nestjs-OpenTelemetry
  • The package is out of date, but there are good insights on how to instrument Nest's components and how to implement decoratos.
• https://github.com/pragmaticivan/nestjs-otel
• https://www.prisma.io/docs/orm/prisma-client/observability-and-logging/opentelemetry-tracing
• https://opentelemetry.io/docs/specs/otel/configuration/sdk-environment-variables/#general-sdk-configuration

[vercel]

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Comments	Updated (UTC)
devtool	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	Oct 31, 2024 1:59pm
manager	🔄 Building (Inspect)	Visit Preview	💬 Add feedback	Oct 31, 2024 1:59pm

[mattschoch]

Looks great!
Agree w/ the comment that using decorators might be nice to reduce the boilerplate, but seems totally fine for now.

[mattschoch]

I'm questioning naming code -- is there a benefit to using ${EntityDataStoreService.name} instead of just "EntityDataStoreService" as a string? When using a raw string instead of interpolation, it makes it easier to do a "find all" in the codebase when you're looking for the line a log was from.

Obv...it tells you the service & method name, so maybe my workflow is what doesn't make sense. But I tend to find something in logs, and want to just copy/paste in "find" to get to the file. wdyt?

I'm fine either way really, just throwing this out in case we all do this..

[wcalderipe]

The reason I opted for ${EntityDataStoreService.name} is to make the span name refactoring proof. Although the side effects are:

• Refactoring the class name will affect the data
• You can't simply copy the span name and look at the whole code base for it

I'm on the fence now.. let me think a bit.

[mattschoch]

Should we use constants for the metric names so that we have one place to look to find all the possible metrics we have?

[wcalderipe]

I would push the metrics discoverability problem to the observability stack rather than trying to solve it in the code.

[mattschoch]

When/how does this get changed?
Will these need to become env configs?

[wcalderipe]

This is the local OTEL collector container configuration. For deploy environments, we'll have to configure it differently.

[mattschoch]

Can we use LGTM locally? If we can use tempo instead of jaeger in the local config, that'd be cool. Not blocking, but might be cool.

[wcalderipe]

Probably. The reason why I opted for Jaeger and the OTEL collector was simplicity for local. Other solutions require you to run 3 to 6 extra containers which increases the cognitive load to understand the system.

[mattschoch]

Should it be enabled by default w/ a flag to disable? Or should it be disabled by default & use the flag to enable?

[wcalderipe]

I'd rather keep it disabled by default because, realistic speaking, we don't need an observability stack to debug locally. Although, log querying and aggregation for development would be nice.