You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
HTTP2 locally-originated stream resets use the NO_ERROR code: When envoy originates a stream reset on an HTTP2 codec it sends a RST_STREAM frame but erroneously uses the NO_ERROR error code, causing clients to think the stream terminated successfully
Description:
When envoy's HTTP2 codec sends a stream reset, it converts the stream reset reason into an HTTP2 error code here. Only two reset reasons are covered, and the remainder reset with NO_ERROR. From the RFC, NO_ERROR indicates:
The associated condition is not a result of an error. For example, a GOAWAY might include this code to indicate graceful shutdown of a connection.
The other reset reasons (see all here) are terminated as NO_ERROR indicating that the stream is complete.
Here's an example where I ran into this issue:
Upstream is HTTP1
Downstream is HTTP2
Downstream sends a GET request as HTTP2
Upstream gets the request headers as HTTP1
Upstream sends happy response headers (200 OK) with Transfer-Encoding: chunked
Downstream receives response headers
Upstream sends response data in chunks, but gets an error part way through the response body. Headers were already sent, so all it can do is reset the HTTP1 stream
Envoy sees this as a protocol error, HPE_INVALID_EOF_STATE
Envoy proxies the reset by resetting the downstream HTTP2 stream
However the downstream stream is reset with 0x00, NO_ERROR
The client sees this and thinks the response body is complete and nothing is amiss, not realizing it doesn't have a complete response
Repro steps:
Python server to provide partial chunked responses over HTTP/1.1:
[2024-09-17 15:54:44.924][54449801][debug][client] [source/common/http/codec_client.cc:170] [Tags: "ConnectionId":"1"] Error dispatching received data: http/1.1 protocol error: HPE_INVALID_EOF_STATE
[2024-09-17 15:54:44.924][54449801][debug][client] [source/common/http/codec_client.cc:107] [Tags: "ConnectionId":"1"] disconnect. resetting 1 pending requests
[2024-09-17 15:54:44.924][54449801][debug][client] [source/common/http/codec_client.cc:158] [Tags: "ConnectionId":"1"] request reset
[2024-09-17 15:54:44.924][54449801][trace][main] [source/common/event/dispatcher_impl.cc:242] item added to deferred deletion list (size=1)
[2024-09-17 15:54:44.924][54449801][debug][router] [source/common/router/router.cc:1323] [Tags: "ConnectionId":"0","StreamId":"164029717455481211"] upstream reset: reset reason: protocol error, transport failure reason:
[2024-09-17 15:54:44.924][54449801][trace][main] [source/common/event/dispatcher_impl.cc:242] item added to deferred deletion list (size=2)
[2024-09-17 15:54:44.924][54449801][trace][main] [source/common/event/dispatcher_impl.cc:242] item added to deferred deletion list (size=3)
[2024-09-17 15:54:44.924][54449801][debug][http] [source/common/http/filter_manager.cc:974] [Tags: "ConnectionId":"0","StreamId":"164029717455481211"] Resetting stream due to upstream_reset_after_response_started{protocol_error}. Prior headers have already been sent
[2024-09-17 15:54:44.925][54449801][debug][http] [source/common/http/conn_manager_impl.cc:243] [Tags: "ConnectionId":"0","StreamId":"164029717455481211"] doEndStream() resetting stream
[2024-09-17 15:54:44.925][54449801][debug][http] [source/common/http/conn_manager_impl.cc:1932] [Tags: "ConnectionId":"0","StreamId":"164029717455481211"] stream reset: reset reason: protocol error, response details: -
[2024-09-17 15:54:44.925][54449801][trace][main] [source/common/event/dispatcher_impl.cc:242] item added to deferred deletion list (size=4)
[2024-09-17 15:54:44.925][54449801][trace][misc] [source/common/event/scaled_range_timer_manager_impl.cc:60] enableTimer called on 0x11e76c2b0 for 3600000ms, min is 3600000ms
[2024-09-17 15:54:44.925][54449801][trace][http2] [source/common/http/http2/codec_impl.cc:1315] [Tags: "ConnectionId":"0"] about to send frame type=3, flags=0
[2024-09-17 15:54:44.925][54449801][trace][http2] [source/common/http/http2/codec_impl.cc:1337] [Tags: "ConnectionId":"0"] send data: bytes=13
[2024-09-17 15:54:44.925][54449801][trace][connection] [source/common/network/connection_impl.cc:529] [Tags: "ConnectionId":"0"] writing 13 bytes, end_stream false
[2024-09-17 15:54:44.925][54449801][trace][http2] [source/common/http/http2/codec_impl.cc:1208] [Tags: "ConnectionId":"0"] sent frame type=3, stream_id=1, length=4
[2024-09-17 15:54:44.925][54449801][debug][http2] [source/common/http/http2/codec_impl.cc:1237] [Tags: "ConnectionId":"0"] sent reset code=0
[2024-09-17 15:54:44.925][54449801][debug][http2] [source/common/http/http2/codec_impl.cc:1362] [Tags: "ConnectionId":"0"] stream 1 closed: 0
Desired behavior:
Envoy should use a non-zero error code when resetting HTTP2 streams. For the chunked encoding case, PROTOCOL_ERROR would seem appropriate.
I have a PR (will submit soon) that should fix this, but it will require us to make some decisions about what the right error codes are for each situation.
The text was updated successfully, but these errors were encountered:
HTTP2 locally-originated stream resets use the NO_ERROR code: When envoy originates a stream reset on an HTTP2 codec it sends a RST_STREAM frame but erroneously uses the NO_ERROR error code, causing clients to think the stream terminated successfully
Description:
When envoy's HTTP2 codec sends a stream reset, it converts the stream reset reason into an HTTP2 error code here. Only two reset reasons are covered, and the remainder reset with NO_ERROR. From the RFC, NO_ERROR indicates:
The other reset reasons (see all here) are terminated as NO_ERROR indicating that the stream is complete.
Here's an example where I ran into this issue:
Transfer-Encoding: chunked
HPE_INVALID_EOF_STATE
Repro steps:
Python server to provide partial chunked responses over HTTP/1.1:
Envoy config:
then reach envoy via an nghttp client (settings, priority, window_update frames omitted):
with partial chunked response:
with complete response:
Envoy trace logs around the event:
Desired behavior:
Envoy should use a non-zero error code when resetting HTTP2 streams. For the chunked encoding case,
PROTOCOL_ERROR
would seem appropriate.I have a PR (will submit soon) that should fix this, but it will require us to make some decisions about what the right error codes are for each situation.
The text was updated successfully, but these errors were encountered: