Add support for HTTP 405 and enhance request handling

.github/copilot-instructions.md

@@ -1,96 +1,7 @@
-# AtomVM HTTPD - AI Coding Guide
+# AtomVM HTTPD — AI Coding Guide
 
-## Project Overview
+This project's agent and AI guidance lives in **[`AGENTS.md`](../AGENTS.md)** at the repo root.
 
-HTTP server library for **AtomVM** - a lightweight Erlang VM for microcontrollers (ESP32, STM32). Mixed Erlang/Elixir codebase using Mix build system.
-
-## Architecture
-
-### Core Components
-```
-gen_tcp_server.erl → httpd.erl → Handler Modules
-     (TCP layer)     (HTTP parsing,   (Request processing)
-                      routing)
-```
-
-- **`gen_tcp_server`**: Generic TCP server behavior wrapping socket operations. Handlers implement `init/1`, `handle_receive/3`, `handle_tcp_closed/2`.
-- **`httpd`**: HTTP 1.1 protocol implementation. Routes requests to handlers based on path prefix matching. Implements `gen_tcp_server` behavior.
-- **Handler behaviors**: Three handler types for different use cases:
-  - `httpd_handler` - Low-level HTTP request handling
-  - `httpd_api_handler` - REST APIs with JSON encoding (implement `handle_api_request/4`)
-  - `httpd_ws_handler` - WebSocket communication (implement `handle_ws_init/3`, `handle_ws_message/2`)
-
-### Handler Return Values
-Handlers return tuples indicating response behavior:
-```erlang
-{reply, Headers, Body, State}   %% Send response, keep connection
-{close, Headers, Body}          %% Send response, close connection
-{noreply, State}                %% Continue accumulating request (streaming)
-{error, not_found | bad_request | internal_server_error}
-```
-
-## Key Patterns
-
-### Path-Based Routing
-Configuration maps URL path prefixes to handler modules:
-```erlang
-Config = [
-    {[<<"api">>], #{handler => httpd_api_handler, handler_config => #{module => MyApi}}},
-    {[<<"ws">>], #{handler => httpd_ws_handler, handler_config => #{module => MyWs}}},
-    {[], #{handler => httpd_file_handler, handler_config => #{app => my_app}}}
-]
-```
-The first matching prefix wins. Path prefix is stripped before passing to handler.
-
-### Creating API Handlers
-Implement `httpd_api_handler` behavior - see `httpd_stats_api_handler.erl`:
-```erlang
--behavior(httpd_api_handler).
--export([handle_api_request/4]).
-
-handle_api_request(get, [<<"endpoint">>], HttpRequest, Args) ->
-    {ok, #{status => <<"ok">>}};  %% Auto-encoded to JSON
-handle_api_request(_Method, _Path, _HttpRequest, _Args) ->
-    not_found.
-```
-
-### Tracing/Debugging
-Enable per-module tracing by uncommenting the define before the include:
-```erlang
--define(TRACE_ENABLED, true).
--include_lib("atomvm_httpd/include/trace.hrl").
-```
-Use `?TRACE("format ~p", [args])` macro. Disabled traces compile to `ok`.
-
-## AtomVM Constraints
-
-- **No hot code loading** - full redeploy required
-- **Limited OTP** - subset of standard library; no `handle_continue`
-- **Memory-sensitive** - prefer binaries, avoid large data structures, use streaming for big payloads
-- **Platform modules**: `atomvm:platform/0`, `esp:*`, `atomvm:read_priv/2` for embedded resources
-
-## Development Commands
-
-```bash
-mix deps.get          # Fetch dependencies
-mix compile           # Build (uses erlc_paths: ["src"])
-mix test              # Run tests (on host Erlang VM, not AtomVM)
-```
-
-Tests run on standard Erlang VM. The `-ifdef(TEST)` guard exposes internal functions for testing.
-
-## Testing Patterns
-Tests use ExUnit. See `test/httpd_integration_test.exs` for socket-level testing:
-- Create test handlers in `test/support/` (added to elixirc_paths in test env)
-- Use `@tag handler_config: %{...}` to customize handler per-test
-- Start server with `:httpd.start_link(port, config)`, test via `:gen_tcp`
-
-## File Organization
-
-| Directory | Purpose |
-|-----------|---------|
-| `src/*.erl` | Erlang source - handlers and core modules |
-| `lib/*.ex` | Elixir source (thin wrapper) |
-| `include/*.hrl` | Header files - HTTP codes, trace macros |
-| `priv/` | Static assets (served via `httpd_file_handler`) |
-| `test/support/` | Test-only modules |
+See it for architecture, handler contracts, routing, commands, testing, and
+repo-specific gotchas (socket setopt allow-list, app-level option stripping,
+send serialization, etc.). Update `AGENTS.md` rather than duplicating guidance here.

.gitignore

@@ -22,4 +22,6 @@ atomvm_httpd-*.tar
 # Temporary files, for example, from tests.
 /tmp/
 
-.elixir_ls/
+.elixir_ls/
+
+mise.local.toml

AGENTS.md

@@ -1,83 +1,182 @@
-# AI Agent Reference for AtomVM HTTPD
+# AtomVM HTTPD — Agent Guide
 
-This project is an HTTP server library for **AtomVM** - a lightweight Erlang/Elixir virtual machine designed for microcontrollers and embedded systems.
+HTTP/WebSocket server **library** for AtomVM (lightweight Erlang VM for
+microcontrollers: ESP32, STM32). Implementation is Erlang (`src/`); `lib/` is a
+thin Elixir convenience wrapper. Built with Mix.
 
-## What is AtomVM?
+## Architecture
 
-AtomVM is a tiny Erlang VM that runs on resource-constrained devices like ESP32, STM32, and other microcontrollers. It allows you to write Erlang and Elixir code for embedded systems.
+```
+gen_tcp_server.erl  →  httpd.erl            →  Handler modules
+(TCP/socket layer)     (HTTP/1.1 parse,         (request processing)
+                        routing, send)
+```
+
+- **`gen_tcp_server`**: generic TCP server behavior wrapping AtomVM `socket`.
+  Implementers provide `init/1`, `handle_receive/3`, `handle_tcp_closed/2`
+  (optional `handle_info/2`). A single gen_server owns the listen socket AND all
+  handler state; per-connection processes only own `recv` and forward data to it.
+  Parsing, dispatch, and the chunked `send` all run in that one gen_server — so
+  responses are serialized across connections (known throughput bottleneck for
+  large/parallel responses).
+- **`httpd`**: HTTP/1.1 protocol + path-prefix routing; implements the
+  `gen_tcp_server` behavior.
+- **Handler behaviors**:
+  - `httpd_handler` — low-level HTTP (`init_handler/2`, `handle_http_req/2`)
+  - `httpd_api_handler` — REST/JSON (`handle_api_request/4`)
+  - `httpd_ws_handler` — WebSocket (`handle_ws_init/3`, `handle_ws_message/2`)
+
+### Handler return values
+
+```erlang
+{reply, Headers, Body, State}   %% send response, keep connection open
+{close, Headers, Body}          %% send response, then close
+{noreply, State}                %% keep accumulating request (streaming)
+{error, not_found | bad_request | internal_server_error}
+```
+
+### Path-based routing
+
+First matching prefix wins; the prefix is stripped before reaching the handler.
 
-- **GitHub**: https://github.com/atomvm/AtomVM
-- **Documentation**: https://www.atomvm.net/doc/master/
-- **Programming Guide**: https://www.atomvm.net/doc/master/programmers-guide.html
+```erlang
+Config = [
+  {[<<"api">>], #{handler => httpd_api_handler, handler_config => #{module => MyApi}}},
+  {[<<"ws">>],  #{handler => httpd_ws_handler,  handler_config => #{module => MyWs}}},
+  {[],          #{handler => httpd_file_handler, handler_config => #{app => my_app}}}
+]
+```
+
+File handler is the catch-all and must be last.
 
-## Key Differences from Standard Erlang/OTP
+### API handler example (see `httpd_stats_api_handler.erl`)
 
-When working on this codebase, keep in mind:
+```erlang
+handle_api_request(get, [<<"endpoint">>], _HttpRequest, _Args) ->
+    {ok, #{status => <<"ok">>}};   %% map auto-encoded to JSON
+handle_api_request(_M, _P, _R, _A) ->
+    not_found.
+```
 
-1. **Limited OTP Support**: AtomVM implements a subset of OTP. Not all standard library modules are available.
+## Commands
+
+- `mix compile` — build (`erlc_paths: ["src"]`).
+- `mix test` — full suite, runs on **host Erlang/OTP**, not AtomVM.
+- `mix test test/httpd_integration_test.exs:NN` — run a single test.
+- `mix format` — Elixir only (`lib`, `test`); does NOT touch Erlang `src/`.
+- `mix deps.get` is a no-op here (no deps). The README dep snippet is for *consumers*.
+- No CI exists; local `mix test` is the only gate.
+
+## Repo gotchas
+
+- **AtomVM `socket` setopt is an allow-list**: only `{socket, reuseaddr|linger|type}`,
+  `{otp, recvbuf}`, `{ip, add_membership}` are supported — **no `{tcp, nodelay}`**.
+  `gen_tcp_server:set_socket_options/2` uses strict `ok = socket:setopt(...)`, so an
+  unsupported key **crashes the server at startup**.
+- **App-level keys are not socket options**: `max_connections` and `chunk_size` ride in
+  the same `SocketOptions` map but are stripped via `maps:without/2` in `init/1` before the
+  setopt fold. Add any new app-level key to that strip list.
+- `socket:send/2` returns `ok | {ok, Rest} | {error, Reason}`; partial sends must be retried
+  (see `try_send_binary/3`).
+- Responses are sent in `chunk_size` slices (default 4096; configurable per server).
+- No `priv/` in this repo; `httpd_file_handler` serves from a *consumer* app's `priv`.
+
+## Testing
+
+- `erlc_options(:test)` injects `{d, TEST}`, enabling `-ifdef(TEST)` internal exports in `httpd.erl`.
+- `test/support/*.ex` (e.g. `TestEchoHandler`) compile only in `:test` (`elixirc_paths`).
+- Integration tests drive the server over raw `:gen_tcp`; per-test handler config via the
+  `handler_config` setup-context key.
+- `TestEchoHandler` replies with its configured `:reply_body` (default `"ok"`) — it does NOT
+  echo the request body.
+
+## Debugging & tracing
+
+Per-module tracing: uncomment the define *before* the include, then use `?TRACE`.
+
+```erlang
+-define(TRACE_ENABLED, true).
+-include_lib("atomvm_httpd/include/trace.hrl").
+```
 
-2. **No `gen_server` callbacks with `handle_continue`**: Some newer OTP features may not be implemented.
+Disabled traces compile to `ok`. Keep default log output quiet — gate noise behind `?TRACE`,
+not `io:format`.
 
-3. **Memory Constraints**: Code should be memory-efficient. Avoid large data structures and prefer streaming/chunked processing.
+## AtomVM constraints
 
-4. **No Hot Code Loading**: Unlike standard Erlang, AtomVM doesn't support hot code swapping.
+- Subset of OTP (no `handle_continue`); no hot code loading (full redeploy required).
+- Memory-sensitive: prefer binaries, stream large payloads, avoid the process dictionary.
+- Platform modules: `atomvm:platform/0`, `esp:*`, `atomvm:read_priv/2`.
 
-5. **Limited Process Dictionary**: Use sparingly if at all.
+## Conventions
 
-6. **Binary Handling**: Binaries are well-supported and preferred for string/data handling.
+- Active development branch is `improvements` (not `main`); README dep examples pin `main`.
+- Erlang `src/` is the source of truth; `lib/atomvm_httpd.ex` is only a convenience wrapper.
 
-## AtomVM-Specific Modules
+## ESP32 Debug Loop (examples/httpd_debug/)
 
-AtomVM provides platform-specific modules:
+A complete debug/test application for iterating on ESP32 hardware with real-world HTTP loads.
 
-- `atomvm:platform/0` - Returns the current platform (e.g., `esp32`, `stm32`, `generic_unix`)
-- `esp:*` - ESP32-specific functions (GPIO, WiFi, etc.)
-- `network:*` - Network configuration for embedded platforms
+### Setup (one-time)
 
-## Project Structure
+1. **WiFi credentials** (compile-time):
+   ```bash
+   export ATOMVM_WIFI_SSID="your-ssid"
+   export ATOMVM_WIFI_PSK="your-password"
+   ```
 
-- `src/` - Erlang source files (.erl)
-- `lib/` - Elixir source files (.ex)
-- `include/` - Header files (.hrl)
-- `priv/` - Static assets and resources
-- `test/` - Test files
+2. **ESP32 connection**: Connect ESP32-S3 to `/dev/ttyACM0` (or update scripts if different port).
 
-## Building
+3. **ESP-IDF environment**: Already available via `get_idf` alias (sources `$HOME/.espressif/v5.5.4/esp-idf/export.sh`). Flash script sources this automatically.
 
-This is a mixed Erlang/Elixir project using Mix:
+### Iteration cycle
 
 ```bash
-mix deps.get
-mix compile
-```
+cd examples/httpd_debug
 
-For deploying to ESP32, you'll need to create an AVM file and flash it.
+# 1. Edit code in src/ (library) or lib/ (test app)
 
-## Useful Tips
+# 2. Build and flash (kills existing monitor, builds, flashes to ESP32)
+./scripts/flash.sh
 
-1. **Debugging**: Use `io:format/2` or `erlang:display/1` for debugging - standard Erlang debugger is not available.
+# 3. Monitor serial output (logs to /tmp/atomvm_serial.log)
+./scripts/monitor.sh
+# Watch for: "HTTPD ready at http://X.X.X.X:80"
 
-2. **Testing Locally**: Tests run on the host machine using standard Erlang VM, not AtomVM itself.
+# 4. Test via browser or automated suite
+open http://X.X.X.X/              # Browser dashboard
+./scripts/test.sh X.X.X.X         # Automated curl tests
 
-3. **Handler Pattern**: This httpd uses a handler-based architecture. Each `*_handler.erl` module handles specific route patterns.
+# 5. Check serial log for crashes/errors
+grep -i "error\|crash\|abort" /tmp/atomvm_serial.log
 
-4. **WebSocket Support**: `httpd_ws_handler.erl` provides WebSocket functionality.
+# 6. Fix and repeat
+```
 
-5. **API Handlers**: Files ending in `_api_handler.erl` are REST API endpoints.
+### What's included
 
-## Common Patterns in This Codebase
+- **Debug API endpoints** (`/api/ping`, `/api/echo`, `/api/generate?size=N`, `/api/memory`):
+  Stress-test request/response sizes up to 1MB. Every request logs heap state to serial.
+- **Built-in stats** (`/api/stats/system`, `/api/stats/memory`): Platform info + ESP32 heap.
+- **Command API** (`/api/cmd/restart`): Restart ESP32 over HTTP.
+- **Browser dashboard** (`/`): Interactive UI for triggering tests, viewing results, monitoring memory.
+- **Automated test suite** (`scripts/test.sh`): Sweeps response sizes (100B → 64KB) and upload sizes (100B → 16KB), reports pass/fail + timing.
 
-### Handler Behavior
+### Tuning parameters
 
-Handlers implement callbacks for HTTP request processing. Check `httpd_handler.erl` for the behavior definition.
+- **`chunk_size`** (default 4096): Set in `lib/httpd_debug.ex` line 13. AtomVM lwIP default send buffer is 8KB; values up to 8192 are safe.
+- **Request timeout** (default 30s): Set via `:httpd.start_link/5` options map (not currently exposed in debug app, but easy to add).
 
-### TCP Server
+### AI agent workflow
 
-`gen_tcp_server.erl` provides the underlying TCP socket management.
+When debugging performance issues or crashes on hardware:
 
-## References
+1. **Flash**: `bash examples/httpd_debug/scripts/flash.sh` (Read tool to check output)
+2. **Monitor**: `bash examples/httpd_debug/scripts/monitor.sh` in background, or Read `/tmp/atomvm_serial.log`
+3. **Extract IP**: Grep serial log for `"HTTPD ready at http://"`
+4. **Test**: `bash examples/httpd_debug/scripts/test.sh <ip>` or individual curl commands
+5. **Analyze**: Read serial log for crash traces (Guru Meditation, stack dumps), parse test failures
+6. **Edit**: Make fixes in `src/` (library code) or `examples/httpd_debug/lib/` (test app)
+7. **Iterate**: Return to step 1
 
-- [AtomVM GitHub](https://github.com/atomvm/AtomVM)
-- [AtomVM Docs](https://www.atomvm.net/doc/master/)
-- [ESP32 Platform Notes](https://www.atomvm.net/doc/master/build-instructions.html#esp32)
-- [Erlang Compatibility](https://www.atomvm.net/doc/master/programmers-guide.html#erlang-compatibility)
+Serial log persists at `/tmp/atomvm_serial.log` across monitor restarts for post-mortem analysis.

README.md

@@ -42,7 +42,7 @@ At a high level, this server supports the following features:
   - Server-initiated push messages
 - **ESP32-optimized networking**
   - Configurable socket options (`SO_REUSEADDR`, buffer sizes, etc.)
-  - 1460-byte send chunking for lwIP compatibility
+  - Configurable send chunking (default 4096 bytes) for lwIP compatibility
   - Incremental I/O list processing to minimize heap pressure
 
 The HTTPd server is designed around a callback architecture, whereby users implement behaviors to handle various requests into the HTTP server. This architecture allows developers to focus on the logic of their applications, as opposed to the nitty gritty details of the HTTP protocol, while still providing access to contextual information about the request, including:
@@ -139,6 +139,10 @@ Supported socket options (per AtomVM's `socket` module):
 - `{otp, recvbuf}` - `non_neg_integer()` - Receive buffer size in bytes
 - `{ip, add_membership}` - Multicast group membership (advanced)
 
+The following keys are handled by `gen_tcp_server` itself and are **not** passed to `socket:setopt`:
+- `max_connections` - `non_neg_integer()` - Maximum concurrent connections (0 = unlimited, default)
+- `chunk_size` - `pos_integer()` - Maximum bytes per `socket:send/2` call (default: `4096`). Tune this to match your platform's lwIP send-buffer headroom. A 100 KB JPEG at 4096 bytes/chunk requires ~25 send calls; at 1460 bytes it required ~70. ESP32 lwIP defaults to an 8 KB send buffer, so values up to 8192 are generally safe.
+
 The default configuration enables `SO_REUSEADDR` which is particularly useful on ESP32 for quick restarts during development.
 
 > Note. The configuration for the HTTPd server is described in more detail below.