Avoid logging and connection backoff in the case of a canceled context

[teyyubismayil]

Avoid logging and connection backoff in the case of a canceled context on the gRPC stream in querier.

What this PR does:
Currently in case of context cancel a lot of unnecessary error logs are produced. Also this pr will potentially improve performance as backoff won't be awaited on context cancel.

Checklist

• Tests updated
• Documentation added
• CHANGELOG.md updated - the order of entries should be [CHANGE], [FEATURE], [ENHANCEMENT], [BUGFIX]

[CLAassistant]

All committers have signed the CLA.

[joe-elliott]

Thanks for the PR! Can you talk about what this is trying to solve? What situations have you seen that are problematic and how does this PR help them?

[teyyubismayil]

@joe-elliott When querier is overloaded and query frontend cancels query I observe a lot of error contacting frontend, received context error while waiting for new LB policy update: context canceled error logs which I assume is not needed. Also I think awaiting backoff is also unnecessary here. Similar conditional exists a few lines after.

[joe-elliott]

So the first error here occurs when there is an issue connecting:

https://github.com/grafana/tempo/blob/main/modules/querier/worker/frontend_processor.go#L74-L79

I think we'd prefer to have backoff here b/c it signals an unreachable or struggling frontend. In this case we'd like queriers to backoff so they don't overwhelm the frontend when it becomes reachable again.

The second part:

https://github.com/grafana/tempo/blob/main/modules/querier/worker/frontend_processor.go#L74-L79

Has special handling for context cancelled b/c this is how the frontend signals that a job is cancelled and tells the querier to stop processing. It's not a great system and I hope to fix it one day.