You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Description:
I have an envoy configuration that uses RDS and has more than 50k routes. Even though I am using DELTA_GRPC sometimes the proxy will end-up not being able to receive any new updates with the error message:
[2024-09-16 17:17:12.814][1][warning][config] [./source/extensions/config_subscription/grpc/grpc_stream.h:190] DeltaVirtualHosts gRPC config stream to xds_cluster closed since 105s ago: 8, grpc: received message larger than max (21173076 vs. 4194304)
Locally, I created a gRCP client with higher grpc.MaxCallRecvMsgSize(math.MaxInt32) and it worked but I am curious if this is something that we want to be able to configure on envoy as well.
Any idea why using the DELTA API is not enough and it batches bigger updates than the gRPC can handle?
I also saw that defaultServerMaxSendMessageSize = math.MaxInt32 vs defaultClientMaxReceiveMessageSize = 1024 * 1024 * 4 which is exactly what causes the issue to appear.
[2024-09-16 17:17:12.814][1][warning][config] [./source/extensions/config_subscription/grpc/grpc_stream.h:190] DeltaVirtualHosts gRPC config stream to xds_cluster closed since 105s ago: 8, grpc: received message larger than max (21173076 vs. 4194304)
The text was updated successfully, but these errors were encountered:
dmavrommatis
changed the title
XDS DeltaVirtualHosts gRPC config stream to xxx closed
XDS DeltaVirtualHosts gRPC config stream to xxx closed: grpc: received message larger than max
Sep 16, 2024
Based on my knowledge of delta XDS, we could probably split up the subscription requests to avoid hitting the default receive message size limit. But that limit is somewhat arbitrary. It don't recall it showing up in the gRPC spec and there's nothing to prevent a different gRPC server implementation from choosing a different limit. I think this ends up being a well-meaning default for servers with untrusted clients that trips up systems with trusted clients as their scale grows.
Based on my knowledge of delta XDS, we could probably split up the subscription requests to avoid hitting the default receive message size limit. But that limit is somewhat arbitrary. It don't recall it showing up in the gRPC spec and there's nothing to prevent a different gRPC server implementation from choosing a different limit. I think this ends up being a well-meaning default for servers with untrusted clients that trips up systems with trusted clients as their scale grows.
I am using the https://github.com/envoyproxy/go-control-plane implementation of xDS server and it looks like it doesn't split up the requests and just full sends all the deltas disregarding the size.
In any case; the 4MB limit size on the receiving end of envoy seems very low. I haven't used/see other implementations of the control-plane (e.g. https://github.com/envoyproxy/java-control-plane) so it might be only the golang one that is the problematic and does not split-up messages.
Title: xDS server sends larger message than max
Description:
I have an envoy configuration that uses RDS and has more than 50k routes. Even though I am using
DELTA_GRPC
sometimes the proxy will end-up not being able to receive any new updates with the error message:Locally, I created a gRCP client with higher
grpc.MaxCallRecvMsgSize(math.MaxInt32)
and it worked but I am curious if this is something that we want to be able to configure on envoy as well.Any idea why using the DELTA API is not enough and it batches bigger updates than the gRPC can handle?
I also saw that
defaultServerMaxSendMessageSize = math.MaxInt32
vsdefaultClientMaxReceiveMessageSize = 1024 * 1024 * 4
which is exactly what causes the issue to appear.Repro steps:
Config:
envoy.yaml
lds.yaml
Logs:
The text was updated successfully, but these errors were encountered: