Jaeger All in one docker gRPC issues

[ashwinhs]

Hi All - Getting this error when i startup jaeger allinone docker latest. Not sure why this is - can anyone help here? I am running this on Windows, Docker for desktop. This is behind a corp proxy, if that's helpful. This is the command i am using to startup

docker run --name jaeger -e COLLECTOR_ZIPKIN_HTTP_PORT=9411 -p 14250:14250 -p 5775:5775/udp -p 6831:6831/udp -p 6832:6832/udp -p 5778:5778 -p 16686:16686 -p 14268:14268 -p 9411:9411 jaegertracing/all-in-one:latest

jaeger_1  | {"level":"warn","ts":1614280472.7859962,"caller":"grpc@v1.29.1/clientconn.go:1275","msg":"grpc: addrConn.createTransport failed to connect to {:14250  <nil> 0 <nil>}. Err: connection error: desc = \"transport: Error while dialing failed to do connect handshake, response: \\\"HTTP/1.0 400 Bad Request\\\\r\\\\nConnection: close\\\\r\\\\n\\\\r\\\\nBad request\\\"\". Reconnecting...","system":"grpc","grpc_log":true}
jaeger_1  | {"level":"info","ts":1614280472.7862206,"caller":"grpc@v1.29.1/clientconn.go:1056","msg":"Subchannel Connectivity change to TRANSIENT_FAILURE","system":"grpc","grpc_log":true}
jaeger_1  | {"level":"info","ts":1614280472.7865677,"caller":"grpc@v1.29.1/clientconn.go:417","msg":"Channel Connectivity change to TRANSIENT_FAILURE","system":"grpc","grpc_log":true}
jaeger_1  | {"level":"info","ts":1614280472.786733,"caller":"grpc/builder.go:119","msg":"Agent collector connection state change","dialTarget":":14250","status":"TRANSIENT_FAILURE"}
jaeger_1  | {"level":"info","ts":1614280473.7868812,"caller":"grpc@v1.29.1/clientconn.go:1056","msg":"Subchannel Connectivity change to CONNECTING","system":"grpc","grpc_log":true}
jaeger_1  | {"level":"info","ts":1614280473.7870724,"caller":"grpc@v1.29.1/clientconn.go:1193","msg":"Subchannel picks a new address \":14250\" to connect","system":"grpc","grpc_log":true}
jaeger_1  | {"level":"warn","ts":1614280473.9255273,"caller":"grpc@v1.29.1/clientconn.go:1275","msg":"grpc: addrConn.createTransport failed to connect to {:14250  <nil> 0 <nil>}. Err: connection error: desc = \"transport: Error while dialing failed to do connect handshake, response: \\\"HTTP/1.0 400 Bad Request\\\\r\\\\nConnection: close\\\\r\\\\n\\\\r\\\\nBad request\\\"\". Reconnecting...","system":"grpc","grpc_log":true}
jaeger_1  | {"level":"info","ts":1614280473.925584,"caller":"grpc@v1.29.1/clientconn.go:1056","msg":"Subchannel Connectivity change to TRANSIENT_FAILURE","system":"grpc","grpc_log":true}
jaeger_1  | {"level":"info","ts":1614280475.517604,"caller":"grpc@v1.29.1/clientconn.go:1056","msg":"Subchannel Connectivity change to CONNECTING","system":"grpc","grpc_log":true}
jaeger_1  | {"level":"info","ts":1614280475.517681,"caller":"grpc@v1.29.1/clientconn.go:1193","msg":"Subchannel picks a new address \":14250\" to connect","system":"grpc","grpc_log":true}
jaeger_1  | {"level":"warn","ts":1614280475.6554408,"caller":"grpc@v1.29.1/clientconn.go:1275","msg":"grpc: addrConn.createTransport failed to connect to {:14250  <nil> 0 <nil>}. Err: connection error: desc = \"transport: Error while dialing failed to do connect handshake, response: \\\"HTTP/1.0 400 Bad Request\\\\r\\\\nConnection: close\\\\r\\\\n\\\\r\\\\nBad request\\\"\". Reconnecting...","system":"grpc","grpc_log":true}
jaeger_1  | {"level":"info","ts":1614280475.6555314,"caller":"grpc@v1.29.1/clientconn.go:1056","msg":"Subchannel Connectivity change to TRANSIENT_FAILURE","system":"grpc","grpc_log":true}
jaeger_1  | {"level":"info","ts":1614280478.3234975,"caller":"grpc@v1.29.1/clientconn.go:1056","msg":"Subchannel Connectivity change to CONNECTING","system":"grpc","grpc_log":true}
jaeger_1  | {"level":"info","ts":1614280478.3236346,"caller":"grpc@v1.29.1/clientconn.go:1193","msg":"Subchannel picks a new address \":14250\" to connect","system":"grpc","grpc_log":true}
jaeger_1  | {"level":"warn","ts":1614280478.5104907,"caller":"grpc@v1.29.1/clientconn.go:1275","msg":"grpc: addrConn.createTransport failed to connect to {:14250  <nil> 0 <nil>}. Err: connection error: desc = \"transport: Error while dialing failed to do connect handshake, response: \\\"HTTP/1.0 400 Bad Request\\\\r\\\\nConnection: close\\\\r\\\\n\\\\r\\\\nBad request\\\"\". Reconnecting...","system":"grpc","grpc_log":true}
jaeger_1  | {"level":"info","ts":1614280478.5105915,"caller":"grpc@v1.29.1/clientconn.go:1056","msg":"Subchannel Connectivity change to TRANSIENT_FAILURE","system":"grpc","grpc_log":true}
jaeger_1  | {"level":"info","ts":1614280481.8225894,"caller":"grpc@v1.29.1/clientconn.go:1056","msg":"Subchannel Connectivity change to CONNECTING","system":"grpc","grpc_log":true}
jaeger_1  | {"level":"info","ts":1614280481.8227663,"caller":"grpc@v1.29.1/clientconn.go:1193","msg":"Subchannel picks a new address \":14250\" to connect","system":"grpc","grpc_log":true}
jaeger_1  | {"level":"warn","ts":1614280482.031132,"caller":"grpc@v1.29.1/clientconn.go:1275","msg":"grpc: addrConn.createTransport failed to connect to {:14250  <nil> 0 <nil>}. Err: connection error: desc = \"transport: Error while dialing failed to do connect handshake, response: \\\"HTTP/1.0 400 Bad Request\\\\r\\\\nConnection: close\\\\r\\\\n\\\\r\\\\nBad request\\\"\". Reconnecting...","system":"grpc","grpc_log":true}

[yurishkuro]

Please try specific version, latest could mean anything, i.e. your Docker could've cached latest from a year ago.

[yurishkuro]

this works fine for me:

docker run --name jaeger1 -e COLLECTOR_ZIPKIN_HTTP_PORT=9411 -p 14250:14250 -p 5775:5775/udp -p 6831:6831/udp -p 6832:6832/udp -p 5778:5778 -p 16686:16686 -p 14268:14268 -p 9411:9411 jaegertracing/all-in-one:1.22

[ashwinhs]

Thanks @yurishkuro I tried all the latest versions, doesnt seem to work for some reason. This is what i could see when i send traces -

{"level":"error","ts":1614284923.5530756,"caller":"grpc/reporter.go:74","msg":"Could not send spans over gRPC","error":"rpc error: code = Unavailable desc = last connection error: connection error: desc = \"transport: Error while dialing failed to do connect handshake, response: \\\"HTTP/1.0 400 Bad Request\\\\r\\\\nConnection: close\\\\r\\\\n\\\\r\\\\nBad request\\\"\"","stacktrace":"github.com/jaegertracing/jaeger/cmd/agent/app/reporter/grpc.(*Reporter).send\n\tgithub.com/jaegertracing/jaeger/cmd/agent/app/reporter/grpc/reporter.go:74\ngithub.com/jaegertracing/jaeger/cmd/agent/app/reporter/grpc.(*Reporter).EmitBatch\n\tgithub.com/jaegertracing/jaeger/cmd/agent/app/reporter/grpc/reporter.go:53\ngithub.com/jaegertracing/jaeger/cmd/agent/app/reporter.(*MetricsReporter).EmitBatch\n\tgithub.com/jaegertracing/jaeger/cmd/agent/app/reporter/metrics.go:83\ngithub.com/jaegertracing/jaeger/cmd/agent/app/reporter.(*ClientMetricsReporter).EmitBatch\n\tgithub.com/jaegertracing/jaeger/cmd/agent/app/reporter/client_metrics.go:120\ngithub.com/jaegertracing/jaeger/thrift-gen/jaeger.(*agentProcessorEmitBatch).Process\n\tgithub.com/jaegertracing/jaeger/thrift-gen/jaeger/agent.go:138\ngithub.com/jaegertracing/jaeger/thrift-gen/jaeger.(*AgentProcessor).Process\n\tgithub.com/jaegertracing/jaeger/thrift-gen/jaeger/agent.go:112\ngithub.com/jaegertracing/jaeger/cmd/agent/app/processors.(*ThriftProcessor).processBuffer\n\tgithub.com/jaegertracing/jaeger/cmd/agent/app/processors/thrift_processor.go:121\ngithub.com/jaegertracing/jaeger/cmd/agent/app/processors.NewThriftProcessor.func2\n\tgithub.com/jaegertracing/jaeger/cmd/agent/app/processors/thrift_processor.go:86"}

It works on jaegertracing/all-in-one:1.11.0 Not sure what changed between 1.11.0 and version upward after that.

[mikeumus]

@ashwinhs don't forget to mark this as the answer! 😉

[jpkrohling]

Same here, the following works for me on Linux (Fedora 33):

$ docker run --name jaeger -e COLLECTOR_ZIPKIN_HTTP_PORT=9411 -p 14250:14250 -p 5775:5775/udp -p 6831:6831/udp -p 6832:6832/udp -p 5778:5778 -p 16686:16686 -p 14268:14268 -p 9411:9411 jaegertracing/all-in-one:1.22.0