Skip to content

test: give each spec file one supertest server instead of one per request - #224

Merged
Bccorb merged 1 commit into
mainfrom
fix/supertest-server-per-file
Aug 30, 2026
Merged

Bccorb merged 1 commit into
mainfrom
fix/supertest-server-per-file

Conversation

@Bccorb

@Bccorb Bccorb commented Aug 30, 2026

Copy link
Copy Markdown
Contributor

Works on #214. Reduces the flake substantially, but does not let me claim the
issue is closed. Numbers and caveats below.

The mechanism, traced rather than guessed

supertest builds a fresh http.Server for every call: the constructor does
http.createServer(app), serverAddress calls app.listen(0) and reads the
port, and end() closes it again. Across this suite that is roughly 440
create/listen/connect/close cycles per run, spread over the worker pool.

I instrumented net.Server.prototype.listen/close,
http.Server.prototype.emit and http.request to record every bind, close,
incoming request, outgoing request, response and client error, then ran the suite
in a loop keeping the trace of any failing run. The run that reproduced this
issue's exact symptom:

FAIL tests/integration/webauthn/webauthn.spec.ts
  > POST /webauthn/register/finish > returns 500 when persisting the credential fails
Error: Test timed out in 20000ms.

sent: 439   served: 438     <- one request never reached any server
binds: 439  closes: 438     <- its server never closed
client responses: 438       client errors: 0
UNSERVED port 51341 /webauthn/register/finish

The client issued the request, no server ever emitted request for it, and
neither side errored. The connection went nowhere silently and the test sat there
until the timeout. The server never closed because supertest closes it in the
response callback, which never fired.

Two theories died on the way, both worth recording so nobody re-runs them:

  • Not a threshold problem. Measured over a full coverage run, the slowest
    test in the suite is 560ms (p50 1.1ms, p99 67ms) against a 20000ms timeout.
    Raising the timeout would only have made the hang slower to surface.
  • Not a wrong-server problem. Over 439 servers in a run: 0 port lifetime
    overlaps, 0 client/server spec mismatches, 0 port reuse. Nothing is ever served
    by a server other than the one its own spec bound. That killed the ~350 call
    site refactor I had been considering.

The change

supertest only calls listen when address() is empty, and only closes a server
it opened itself. Handing it a server that is already listening therefore skips
the whole cycle: one server per app, for the life of the spec file. listen(0)
binds synchronously (listening is true and address() populated on return), so
there is no half-bound server to hand over.

Doing that per spec would mean threading a server through ~350 call sites.
Instead tests/setup/supertestServer.ts wraps supertest's default export once, in
the shared setup, swapping an app for a cached listening server and closing them
in afterAll. No spec changes at all. Binds per run drop from 439 to 39.

What it measurably does, and does not do

Same command, same machine, before and after:

Configuration Before After
npm run test:run, 80 runs failures observed 0 failures
npm run test:run with the tracer attached, 80 runs 10 failures 2 failures
npm run coverage, 20 runs (see #214, 1 in 60) see below

The tracer does synchronous file writes around listen, connect and close, so it
amplifies the race and is a sensitive detector rather than a realistic
configuration.

The honest caveat. Of the 2 remaining traced failures, one is still a 20
second timeout. Breaking the traced numbers down by class:

Class Before (traced, 80 runs) After (traced, 80 runs)
Wrong status (404/401 where the route and mocks are fine) 9 1
20 second timeout 1 1

So this change clearly reduces overall flakiness, but it does not
demonstrably eliminate the timeout. Either the residual is a second path to the
same symptom, or the sample is too small to separate 1 from 1. I would rather say
that than claim a fix I have not shown.

The tracer also became unreliable once the fix was in: post-fix it reports whole
files as "unserved" while most of their tests demonstrably pass, so it
under-records served in the shared-server regime. Its post-fix counts are
indicative only, which is why the untraced runs above are the ones I would weigh.

A second, separate mechanism

The wrong-status class is not the lost-request mechanism. Pre-fix traces show
those responses were genuinely produced by the correct app (0 ghosts, 0
mismatches), so they are real answers rather than misrouted ones. POST /login
answering 404 is the striking one, since src/controllers/authentication.ts has
no 404 branch at all. That wants its own ticket; this change appears to reduce it
but was not aimed at it and I cannot explain why it would.

Not user facing

Test infrastructure only, so no changeset. Nothing under src/ changes.

…uest

supertest builds a fresh http.Server for every call: the constructor does
http.createServer(app), serverAddress calls app.listen(0) and reads the port,
and end() closes it again. Across this suite that is roughly 440
create/listen/connect/close cycles per run, spread over the worker pool.

Tracing every bind, close, incoming request, outgoing request and client
response caught a run reproducing the timeout in #214: 439 requests sent against
438 served, and 439 binds against 438 closes, with 438 client responses and zero
client errors. One request was issued, no server ever received it, and neither
side errored. The connection went nowhere silently and the test sat there until
the 20 second timeout. Its server never closed because supertest closes it in
the response callback, which never fired.

Handing supertest a server that is already listening skips that cycle. It only
calls listen when address() is empty, and only closes a server it opened itself,
so one server per app survives the whole file. listen(0) binds synchronously, so
there is no half-bound server to hand over. Binds per run drop from 439 to 39.

Wrapped in the shared setup rather than in each spec, because threading a server
through the roughly 350 call sites is a lot of churn for the same result.

Measured on the same machine, same commands. 80 runs of npm run test:run: no
failures. 20 runs of npm run coverage: no failures, which is the acceptance this
issue asked for. With the tracer attached, which amplifies the race with
synchronous writes around listen and close, 80 runs went from 10 failures to 2.

It does not close #214. One of those two remaining traced failures is still a 20
second timeout, so the reduction is clear but the timeout is not demonstrably
gone. The rest were wrong statuses on routes that exist, which the traces show
were genuine answers from the correct app rather than misrouted ones, and which
belong to a separate mechanism.

Refs #214
@Bccorb
Bccorb merged commit a512aec into main Aug 30, 2026
4 checks passed
@Bccorb
Bccorb deleted the fix/supertest-server-per-file branch August 30, 2026 23:38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant