Repository navigation
HTTP/2 Data padding is not counted toward the connection flow-control window. This deadlocks connections with peers that pad. #9772
Description
Summary
Http2Reader does not count DATA frame padding toward the connection-level flow-control window. When the peer pads its DATA frames, the server's view of the connection window drains faster than the client's counter grows, the server's window reaches zero before the client reaches its next WINDOW_UPDATE threshold, and the connection deadlocks. PING/PONG keeps working, so the connection stays in the pool and every subsequent call on it times out.
Observed on 5.5.0; the code is unchanged on main.
Where
Http2Reader.readData reads the pad-length byte, computes the body length without padding, calls handler.data(..., dataLength) with the body length only, and then skips the padding:
Http2Connection.updateConnectionFlowControl is therefore only ever fed the body length, both through the stream path (Http2Stream.FramingSource.receive) and the unknown-stream path:
The pad-length byte and the padding bytes are consumed from the socket but never credited back to the peer.
Spec
RFC 9113 §6.1: "The entire DATA frame payload is included in flow control, including the Pad Length and Padding fields if present."
https://www.rfc-editor.org/rfc/rfc9113.html#section-6.1
Go's net/http2 fixed the same bug in 2016 and refunds padding immediately on receipt (golang/go#16556, https://go.googlesource.com/net/+/35028a49ca5a73b486af60cd20ac21cd6b67bfdb). Google's quiche fixed a padding flow-control leak that stalled Envoy connections (envoyproxy/envoy#32401, https://quiche.googlesource.com/quiche/+/9e875938052443c7f51b25012093be59e20598ac). Netty charges data.readableBytes() + padding.
How it shows up in practice
Google Front End pads every response DATA frame with a random 0–255 bytes. A production client of ours talking to an API behind GFE deadlocked every long-lived connection at the same point. Measured with a FlowControlListener on the pool:
| response body | ~250 B |
| DATA frame on the wire | ~372 B average, uniform 247–503 B (`DATA END_STREAM |
| fraction the client counts | ~0.665 |
| client window | 16 MiB advertised, WINDOW_UPDATE of 8 MiB once 8 MiB has been counted |
| server bytes sent at the freeze | 24.0 MiB (16 MiB initial + one 8 MiB grant) |
WindowCounter at the freeze |
total 15.7–16.7 MiB, acknowledged 8,388,6xx: below the threshold for the second grant |
| responses per connection before the freeze | ~67,000 |
After the freeze the only inbound frames were PINGs and the server's own WINDOW_UPDATEs for our request bytes. No HEADERS or DATA, no GOAWAY, no RST_STREAM. Http2Connection.isHealthy() kept returning true, so the connection was reused for every call until the server eventually closed it.
Failing test
The Http2Writer used by the test peer cannot write padded frames, so the test writes DATA frames by hand: a 9-byte frame header with FLAG_PADDED, one pad-length byte, the body, then the padding. Send enough frames for the padded total to exceed half the client window, then read the WINDOW_UPDATE the client sends on stream 0. Expected: the increment equals the padded bytes received. Actual on 5.5.0: the increment equals the body bytes only. (I can turn this into a test in Http2ConnectionTest if that helps.)
Workaround
Restricting the affected client to HTTP/1.1 removes the problem entirely.
Activity
Amir-Golmoradi commentedon Oct 8, 2026
Hi, I’d like to work on this issue.
I’ve reviewed the report and the relevant HTTP/2 flow-control path. This looks like a real bug in how padded DATA frames are accounted for against the connection window, and I’d like to investigate the root cause, add a focused regression test, and implement the fix.
If no one is currently working on it, could you please assign this issue to me?
dimitris-selal commentedon Oct 11, 2026
Nobody is working on it. We worked around it on our side by forcing HTTP/1.1 for that client, so a proper fix is all yours, go ahead. I can’t assign (read access only), a maintainer will have to.
ashish28yadav commentedon Oct 12, 2026
Hi! I've been exploring this issue and would like to contribute to the fix. I understand that the key problem is accounting for the full padded DATA frame payload in connection-level flow control.
I plan to study the existing flow-control implementation, reproduce the issue with a focused regression test, and verify that padding bytes are accounted for correctly without double-counting them at the stream level.
I noticed that Amir-Golmoradi has already expressed interest, so I'd be happy to coordinate rather than duplicate work. Is there a preferred next step for contributing to this fix?
Thank you!
Amir-Golmoradi commentedon Oct 12, 2026
Hi again, I've reproduced this and opened a PR with a fix and tests: #9786
The fix reports the discarded padding (including the Pad Length field) from
Http2Readerthrough a newHandler.dataPadding()callback, and credits it to the connection window and, when the stream exists, to the stream window. It also covers DATA frames for unknown/removed streams and pushed streams.The tests cover the padded-frame cases from your report, so no need to write them unless you'd like to compare approaches. I'm also happy to split out the stream-level part if a smaller change is preferred.
dimitris-selal commentedon Oct 14, 2026
Tested this branch (983d9f6) against the server from #9772, an HTTPS API behind Google Front End that pads almost every DATA frame. One HTTP/2 connection, about 100 requests/s, 120k requests, counting inbound DATA payload and outbound connection WINDOW_UPDATEs from OkHttp’s frame log.
okhttp 5.5.0: stalled after 36,069 responses, at exactly 16.0 MiB received, the server’s initial window. No WINDOW_UPDATE had been sent after the preface. Every request after that timed out.
This branch: all 120,000 responses, no errors, same single connection, 41 MiB received. WINDOW_UPDATEs went out every 8 MiB of padded payload.
The fix works.
Amir-Golmoradi commentedon Oct 14, 2026
Hi @dimitris-selal , hope you're doing well.
Thanks for testing it, I'm happy that it works