Instrumenting Kubernetes operator for tracing

[Rajarajan97]

We have seen examples of tracing for request and response scenario. But majority of work is done in backend in separate like Kubernetes custom controller(operator). How do we instrument a full operator flow.Because the operator workflow might be divided into different steps. Each time it will run one step. How to instrument it such that, we can see the whole tracing from beginning to end

@joe-elliott @electron0zero

[mapno]

If I understand correctly, and you want traces from your operator, you'll need instrumentation in the operator itself. If you're writing your own operators, you can use of the OpenTelemetry SDKs for instrumentation

[Rajarajan97]

I'm writing my own operator. We have a workflow that we have broken it into multiple steps in the operator(using switch case or if blocks).We store the current step in one of the fields in CRD. it will complete one step and reconcile it again to the operator. when it runs again it will go into the next block. How do I propagate the trace/span context in this flow? If I pick the primary trace start, it should have all the steps ran in the workflow. How can we do that using telemetry?

@mapno

[joe-elliott]

This seems like a question about your code structure which it's tough to give advice on. You would likely to hold a go context and pass it to the individual steps as they are executing. That's easy to say and may be harder to accomplish.