Unable to pause request or receive dispatch callback during OnHttpRequestBody

[prodion23]

Describe the bug / error

Hello 👋
Wondering if this is an expected/known limitation, but I'm having an issue blocking the request from going to the upstream during OnHttpRequestBody.

Calling proxywasm.DispatchHttpCall and returning Action.Pause during OnHttpRequestBody does not stop the request from going to the upstream.

At the bottom of this issue I've pasted a small repro example filter + envoy logs showing the issue.

I've tried similar code in OnHttpRequestHeaders and I'm successfully able to wait on the dispatched call to return.

What is your Envoy/Istio version?

Envoy: 1.21.1 2022-02-21

What is the SDK version?

v0.17.0

TinyGo Version:

 tinygo version
tinygo version 0.22.0 darwin/amd64 (using go version go1.17.4 and LLVM version 13.0.0)

Example filter showing issue

package main

import (
	"github.com/tetratelabs/proxy-wasm-go-sdk/proxywasm"
	"github.com/tetratelabs/proxy-wasm-go-sdk/proxywasm/types"
)

var (
	Version string
)

type vmContext struct {
	// Embed the default VM context here,
	// so that we don't need to reimplement all the methods.
	types.DefaultVMContext
}

type pluginContext struct {
	// Embed the default plugin context here,
	// so that we don't need to reimplement all the methods.
	types.DefaultPluginContext
}

type httpContext struct {
	// Embed the default http context here,
	// so that we don't need to reimplement all the methods.
	types.DefaultHttpContext
	contextID   uint32
	bodyCapSize int
	url         string
}

func main() {
	proxywasm.SetVMContext(&vmContext{})
}

// Override types.DefaultVMContext.
func (*vmContext) NewPluginContext(contextID uint32) types.PluginContext {
	return &pluginContext{}
}

// Override types.DefaultPluginContext.
func (ctx *pluginContext) NewHttpContext(contextID uint32) types.HttpContext {
	return &httpContext{
		contextID: contextID,
	}
}

// Override types.DefaultHttpContext.
func (ctx *httpContext) OnHttpRequestHeaders(numHeaders int, endOfStream bool) types.Action {
	// we need to extract url for response capture, even if request capture is disabled
	//try go get scheme header first
	scheme, _ := proxywasm.GetHttpRequestHeader(":scheme")
	// if scheme not present, get x-forwarded-proto
	if scheme == "" {
		scheme, _ = proxywasm.GetHttpRequestHeader("x-forwarded-proto")
	}

	host, _ := proxywasm.GetHttpRequestHeader(":authority")

	path, _ := proxywasm.GetHttpRequestHeader(":path")
	if scheme != "" && host != "" && path != "" {
		ctx.url = scheme + "://" + host + path
		proxywasm.LogDebugf("Captured url: %s", ctx.url)
	}

	return types.ActionContinue
}

func (ctx *httpContext) OnHttpRequestBody(bodySize int, endOfStream bool) types.Action {
	ctx.bodyCapSize += bodySize
	if !endOfStream {
		proxywasm.LogInfo("returning pause OnHttpRequestBody")
		return types.ActionPause
	}
	proxywasm.LogInfo("Processing with full HttpRequestBody")

	bodyData, err := proxywasm.GetHttpRequestBody(0, ctx.bodyCapSize)
	if err != nil {
		proxywasm.LogErrorf("failed to get request body: %v", err)
	}

	duringReqBodyCallback := func(numHeaders, bodySize, numTrailers int) {
		proxywasm.LogInfo("In duringReqBodyCallback")
		hs, err := proxywasm.GetHttpCallResponseHeaders()
		if err != nil {
			proxywasm.LogCriticalf("failed to get response body: %v", err)
			return
		}

		for _, h := range hs {
			proxywasm.LogInfof("response header from %s: %s: %s", "some-local-cluster", h[0], h[1])
		}

		b, err := proxywasm.GetHttpCallResponseBody(0, bodySize)
		if err != nil {
			proxywasm.LogCriticalf("failed to get response body: %v", err)
			proxywasm.ResumeHttpRequest()
			return
		}
		proxywasm.LogInfof("check req body: %v", string(b))
		// do something with req body & check a field
		err = proxywasm.ResumeHttpRequest()

		if err != nil {
			return
		}
	}

	headers := make([][2]string, 5)
	headers[0] = [2]string{"content-type", "text/plain"}
	headers[1] = [2]string{":method", "POST"}
	headers[2] = [2]string{":path", "/my_endpoint"}
	headers[3] = [2]string{":authority", "localhost"}
        headers[4] = [2]string{"http.url", ctx.url}

	proxywasm.LogError("About to dispatch call")
	_, err = proxywasm.DispatchHttpCall("some-local-cluster", headers, bodyData, nil, 5000, duringReqBodyCallback)
	if err != nil {
		proxywasm.LogErrorf("failed to dispatch extension call: %v", err)
	}

	proxywasm.LogInfo("Returning ActionPause from OnHttpRequestBody - end of method - expecting callback to be called")
	return types.ActionPause
}


func (ctx *httpContext) OnHttpResponseHeaders(numHeaders int, endOfStream bool) types.Action {
	proxywasm.LogInfo("In OnHttpResponseHeader call")
	return types.ActionContinue
}

Envoy logs when I send a request:

[2022-04-29 14:06:56.899][1489852][info][wasm] [source/extensions/common/wasm/context.cc:1172] wasm log: returning pause OnHttpRequestBody
[2022-04-29 14:06:56.899][1489852][info][wasm] [source/extensions/common/wasm/context.cc:1172] wasm log: Processing with full HttpRequestBody
[2022-04-29 14:06:56.899][1489852][error][wasm] [source/extensions/common/wasm/context.cc:1178] wasm log: About to dispatch call
[2022-04-29 14:06:56.900][1489852][info][wasm] [source/extensions/common/wasm/context.cc:1172] wasm log: Returning ActionPause from OnHttpRequestBody - end of method - expecting callback to be called
[2022-04-29 14:06:56.902][1489852][info][wasm] [source/extensions/common/wasm/context.cc:1172] wasm log: In OnHttpResponseHeader call

Note that the call is dispatched, but the callback function is never invoked & the request continues to the upstream application.

[mathetake]

This is the limitation of the current implementation in Envoy/Prxoy-Wasm C++ host. See for example proxy-wasm/proxy-wasm-cpp-host#143 proxy-wasm/proxy-wasm-cpp-sdk#62

I don't think this is actionable from SDK side, so I'm closing this for now.

Anyway thank you for rasigin this up!