Websocket support for search streaming

[joe-elliott]

What this PR does:
Adds websocket support for streaming and deprecates the GRPC endpoint. Due to the issues we've seen with sharing GRPC and HTTP on the same port (which Grafana requires) we believe this is a better path for streaming data. The way this is currently implemented Tempo will close the websocket after it has completed the query and ignore all messages from the client except for normal closure (1000).

Other changes:

• Fixes a bug where spss is not respected on the GRPC endpoint
• Extended the tempo search cli to support normal http, streaming websockets and streaming grpc

Which issue(s) this PR fixes:
Fixes #2917

Checklist

• Tests updated
• Documentation added
• CHANGELOG.md updated - the order of entries should be [CHANGE], [FEATURE], [ENHANCEMENT], [BUGFIX]

[knylander-grafana]

Approving doc changes. Thank you for adding docs.

[mdisibio]

Looking pretty gtm. Still reviewing and hope to test it out locally.

[electron0zero]

lgtm.

ps: I didn't test it localy, I am trusting @mdisibio to do it :)

[mdisibio]

Testing this locally and looking great. However noticed a small issue with the closure error handling.

Debugging shows that Tempo is finishing the query successfully and closing the connection from the server-side on line 210 However then the deferred reader to detect unexpected client closure is tripping. Instead of getting websocket.CloseNormalClosure it's getting a net.OpError wrapping a poll.errNetClosing. The end result is it logs the error each time. Not noticing any other impact.

Reproduced with both:

• ./tempo-cli query api search "localhost:3200" "{duration>1s}" 2023-10-03T00:00:00Z 2023-10-06T00:00:00Z --limit=10000 --use-ws
• curl -v "ws://localhost:3200/api/search-ws?start=1696305600&end=1696564800&limit=100000&q=%7Bduration%3E1s%7D"

[joe-elliott]

Testing this locally and looking great. However noticed a small issue with the closure error handling.

Good catch. Not sure how I missed this. I feel like the cleanest solution is just to store a bool on the server side that records whether or not we've attempted a closure on the connection. I don't think the error produced can be tested directly to determine if it was graceful or unexpected: 2d4206c