How to Ignore context.Canceled Error

[Ajilagi]

I have a service that uses context cancelation by design to stop any previous process if the other/newest one is coming. The way I make a span is exactly similar to the any example from the instrumentation such as below

• https://github.com/open-telemetry/opentelemetry-go-contrib/tree/main/instrumentation/net/http/otelhttp/example
• https://github.com/open-telemetry/opentelemetry-go-contrib/tree/main/instrumentation/google.golang.org/grpc/otelgrpc/example
• ...

Mostly, the span starts from the http server wrapper, and then it produces children span from the http client wrapper, grpc client wrapper, or sql client wrapper ( I didn't manually start/end any parent/children span)

My problem is when it comes to my instrumentation such as otelhttp, otelgrpc, otelsql, etc. The instrumentation always produces a span with the otel.status: 'ERROR' and it leads to an increased error rate.

Is there any way to replace/modify any field by configuring the tracer/trace provider/span processor or anything on a specific service? I don't want to add any changes to the collector, because it is only an exception in one of my services.

I've found a possible way of using the SpanProcessor, the idea is I could use the OnEnd func to filter the context.Canceled error. But, there's an issue because the OnEnd function only receive the ReadOnlySpan. What I want to do is to get rid of the otel.status: 'ERROR' field, not to drop the entire span. Are there any ideas to solve my usecase?

opentelemetry-go/sdk/trace/span_processor.go

Lines 15 to 50 in b5a9cfb

	type SpanProcessor interface {
	// DO NOT CHANGE: any modification will not be backwards compatible and
	// must never be done outside of a new major release.
	// OnStart is called when a span is started. It is called synchronously
	// and should not block.
	OnStart(parent context.Context, s ReadWriteSpan)
	// DO NOT CHANGE: any modification will not be backwards compatible and
	// must never be done outside of a new major release.
	// OnEnd is called when span is finished. It is called synchronously and
	// hence not block.
	OnEnd(s ReadOnlySpan)
	// DO NOT CHANGE: any modification will not be backwards compatible and
	// must never be done outside of a new major release.
	// Shutdown is called when the SDK shuts down. Any cleanup or release of
	// resources held by the processor should be done in this call.
	//
	// Calls to OnStart, OnEnd, or ForceFlush after this has been called
	// should be ignored.
	//
	// All timeouts and cancellations contained in ctx must be honored, this
	// should not block indefinitely.
	Shutdown(ctx context.Context) error
	// DO NOT CHANGE: any modification will not be backwards compatible and
	// must never be done outside of a new major release.
	// ForceFlush exports all ended spans to the configured Exporter that have not yet
	// been exported. It should only be called when absolutely necessary, such as when
	// using a FaaS provider that may suspend the process after an invocation, but before
	// the Processor can export the completed spans.
	ForceFlush(ctx context.Context) error
	// DO NOT CHANGE: any modification will not be backwards compatible and
	// must never be done outside of a new major release.
	}

[XSAM]

A newly defined OnEnding method for SpanProcessor from open-telemetry/opentelemetry-specification#4024 seems to fulfill such a use case, though we haven't implemented this.

I am interested in implementing this method if other @open-telemetry/go-maintainers are OK with adopting this at an early stage.