Repository navigation
test: give each spec file one supertest server instead of one per request - #224
Merged
Merged
Conversation
…uest supertest builds a fresh http.Server for every call: the constructor does http.createServer(app), serverAddress calls app.listen(0) and reads the port, and end() closes it again. Across this suite that is roughly 440 create/listen/connect/close cycles per run, spread over the worker pool. Tracing every bind, close, incoming request, outgoing request and client response caught a run reproducing the timeout in #214: 439 requests sent against 438 served, and 439 binds against 438 closes, with 438 client responses and zero client errors. One request was issued, no server ever received it, and neither side errored. The connection went nowhere silently and the test sat there until the 20 second timeout. Its server never closed because supertest closes it in the response callback, which never fired. Handing supertest a server that is already listening skips that cycle. It only calls listen when address() is empty, and only closes a server it opened itself, so one server per app survives the whole file. listen(0) binds synchronously, so there is no half-bound server to hand over. Binds per run drop from 439 to 39. Wrapped in the shared setup rather than in each spec, because threading a server through the roughly 350 call sites is a lot of churn for the same result. Measured on the same machine, same commands. 80 runs of npm run test:run: no failures. 20 runs of npm run coverage: no failures, which is the acceptance this issue asked for. With the tracer attached, which amplifies the race with synchronous writes around listen and close, 80 runs went from 10 failures to 2. It does not close #214. One of those two remaining traced failures is still a 20 second timeout, so the reduction is clear but the timeout is not demonstrably gone. The rest were wrong statuses on routes that exist, which the traces show were genuine answers from the correct app rather than misrouted ones, and which belong to a separate mechanism. Refs #214
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Works on #214. Reduces the flake substantially, but does not let me claim the
issue is closed. Numbers and caveats below.
The mechanism, traced rather than guessed
supertest builds a fresh
http.Serverfor every call: the constructor doeshttp.createServer(app),serverAddresscallsapp.listen(0)and reads theport, and
end()closes it again. Across this suite that is roughly 440create/listen/connect/close cycles per run, spread over the worker pool.
I instrumented
net.Server.prototype.listen/close,http.Server.prototype.emitandhttp.requestto record every bind, close,incoming request, outgoing request, response and client error, then ran the suite
in a loop keeping the trace of any failing run. The run that reproduced this
issue's exact symptom:
The client issued the request, no server ever emitted
requestfor it, andneither side errored. The connection went nowhere silently and the test sat there
until the timeout. The server never closed because supertest closes it in the
response callback, which never fired.
Two theories died on the way, both worth recording so nobody re-runs them:
test in the suite is 560ms (p50 1.1ms, p99 67ms) against a 20000ms timeout.
Raising the timeout would only have made the hang slower to surface.
overlaps, 0 client/server spec mismatches, 0 port reuse. Nothing is ever served
by a server other than the one its own spec bound. That killed the ~350 call
site refactor I had been considering.
The change
supertest only calls
listenwhenaddress()is empty, and only closes a serverit opened itself. Handing it a server that is already listening therefore skips
the whole cycle: one server per app, for the life of the spec file.
listen(0)binds synchronously (
listeningis true andaddress()populated on return), sothere is no half-bound server to hand over.
Doing that per spec would mean threading a server through ~350 call sites.
Instead
tests/setup/supertestServer.tswraps supertest's default export once, inthe shared setup, swapping an app for a cached listening server and closing them
in
afterAll. No spec changes at all. Binds per run drop from 439 to 39.What it measurably does, and does not do
Same command, same machine, before and after:
npm run test:run, 80 runsnpm run test:runwith the tracer attached, 80 runsnpm run coverage, 20 runsThe tracer does synchronous file writes around listen, connect and close, so it
amplifies the race and is a sensitive detector rather than a realistic
configuration.
The honest caveat. Of the 2 remaining traced failures, one is still a 20
second timeout. Breaking the traced numbers down by class:
So this change clearly reduces overall flakiness, but it does not
demonstrably eliminate the timeout. Either the residual is a second path to the
same symptom, or the sample is too small to separate 1 from 1. I would rather say
that than claim a fix I have not shown.
The tracer also became unreliable once the fix was in: post-fix it reports whole
files as "unserved" while most of their tests demonstrably pass, so it
under-records
servedin the shared-server regime. Its post-fix counts areindicative only, which is why the untraced runs above are the ones I would weigh.
A second, separate mechanism
The wrong-status class is not the lost-request mechanism. Pre-fix traces show
those responses were genuinely produced by the correct app (0 ghosts, 0
mismatches), so they are real answers rather than misrouted ones.
POST /loginanswering 404 is the striking one, since
src/controllers/authentication.tshasno 404 branch at all. That wants its own ticket; this change appears to reduce it
but was not aimed at it and I cannot explain why it would.
Not user facing
Test infrastructure only, so no changeset. Nothing under
src/changes.