Skip to content

Decode chunked request bodies before passing them to httpbin - #101

Open
pylaterreur wants to merge 1 commit into
kevin1024:masterfrom
pylaterreur:decode-chunked-request-bodies
Open

pylaterreur wants to merge 1 commit into
kevin1024:masterfrom
pylaterreur:decode-chunked-request-bodies

Conversation

@pylaterreur

Copy link
Copy Markdown

Fixes #33.

Chunked uploads don't work against the local server: requests.post(httpbin + "/post", data=some_generator) gets a 501 instead of an echo. wsgiref doesn't decode chunked request bodies. httpbin answers chunked requests with a 501, without reading them, unless it runs under gunicorn, which decodes them for it (see its before_request hook). The server then closes the connection, so a client still sending the body gets a broken pipe instead of the 501.

That broken pipe is why VCR.py's tests for chunked uploads fail now and then. kevin1024/vcrpy#1002 tried to work around it in VCR.py, but as noted there, the limitation is in the wsgiref server pytest-httpbin uses.

Fix

When a request has Transfer-Encoding: chunked, Handler now reads and decodes the body itself, and gives it to the application with a Content-Length, like any other request. Chunk extensions and trailer fields are ignored, and a malformed chunked body gets a 400.

As a result, httpbin echoes the body of chunked uploads, and reports a Content-Length header instead of Transfer-Encoding.

Tests

  • New tests in tests/test_server.py, all of which fail on master:
    • a chunked upload over HTTP and HTTPS gets a 200, with its body echoed;
    • so does a chunked upload that pauses between chunks, the case that used to break the pipe;
    • a malformed chunk size gets a 400.
  • The whole suite passes on Linux on CPython 3.8.20, 3.9.25, 3.10.21, 3.11.16, 3.12.14, 3.13.16, 3.14.8, 3.15.0rc2 and 3.15.0rc2t, and on PyPy 3.11.16. The new tests passed 50 runs out of 50 on CPython 3.13 and PyPy 3.11.
  • CI on my fork: all 24 jobs pass on Linux, macOS and Windows. The existing test_dont_crash_on_handshake_timeout failed once on PyPy 3.11 on macOS, then passed when re-run. Its captured log line lacked the final newline: the test reads the output as soon as the server closes the socket, possibly before the server thread has finished printing. This change doesn't touch that test or the code it covers.
  • VCR.py's three chunked-upload tests failed in 23 of 100 runs with pytest-httpbin 2.1.0, and in none of 100 with this change (CPython 3.13, same load). VCR.py's whole suite passes with it.
  • pre-commit: all hooks pass on the changed files, except flake8's E231 on the existing f-string in Server.url. flake8 6.0.0 reports the same on master when run on Python 3.12 or later.

🤖 Generated with Claude Code

https://claude.ai/code/session_01GwUvhmzgfkiZ6vo4uGyesN

wsgiref doesn't decode chunked request bodies, and httpbin answers them
with a 501 without reading them, unless it runs under gunicorn. So a
chunked upload got a 501 instead of an echo, and since the server then
closed the connection, a client still sending the body got a broken
pipe.

Decode the body in Handler, and give it to the application with a
Content-Length. A malformed chunked body gets a 400.

Fixes kevin1024#33

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GwUvhmzgfkiZ6vo4uGyesN
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Streamed uploads fail against local httpbin

1 participant