Header-only response will not trigger resp(rpc) handling

[jokerwyt]

• Backend: envoy native

# circuitbreaker.appnet
state:
    max_concurrent_req: int
    pending_req: int
    drop_count: int

init():
    max_concurrent_req = 1
    pending_req = 0
    drop_count = 0

req(rpc):
    match pending_req <= max_concurrent_req:
	true =>
            pending_req = pending_req + 1
	    send(rpc, Down)
	false =>
            drop_count = drop_count + 1
	    send(err('circuit breaker'), Up)

resp(rpc):
    pending_req = pending_req - 1
    send(rpc, Up)

We put the whole resp() into Envoy filter's encodeData(). When there is a header-only response, the response handling will NOT be executed as expected.

Our current server will not return header-only responses for now. But if we have chain like circuitbreaker -> firewall, when the firewall rejects the request, the circuitbreaker will receive a header-only response. Then it will skip the resp handling that should have been executed, which makes pending_req not consistent.

Possible solutions

As background, our current appnet program semantics are:

if we send an error in req() in THIS element, we do not execute resp handling. But if we normally pass the request downstream, we should. (even the later elements return error directly) .

(But some of our appnet programs mistakenly assume we are always handling successful responses in resp, which will be another problem to fix.)

We have two directions:

1. Separate header and body handling of resp(). This is closer to Envoy Filter itself and it demands more verbose handling.
2. Find a way to detect header-only responses in runtime, and do resp() things in encodeHeader if so. This will introduce minimal engineering effort. The error caused by access to body data when impossible is left to users.