[SQD-Trevor-Agent]

Problem

The hl-node-fills historical-data archive stalled: its ingest cursor froze at block 1183474774 (data timestamp 2026-10-14 11:59:59 UTC) and lag grew unbounded, firing ArchiveIngestLagging (30% behind 24h baseline).

The ingest-hl-node-fills pod runs this hyperliquid-fills-ingest service as a sidecar on localhost:3000; the data-sqd-archive writer streams blocks from it. From ~13:11 UTC the writer logged, exactly every ~125s:

data streaming error ... connection closed before message completed

and the ingest sidecar logged, at the matching cadence:

reading 20261014/12.lz4
Error: output stream is no longer writable   (waitDrain -> HyperliquidIngest.ingest)

Root cause

Hyperliquid fills volume surged during hour 12. The upstream S3 chunk node_fills_by_block/hourly/20261014/12.lz4 is 89 MB compressed / 439 MB decompressed — ~3× the preceding chunks (11.lz4 = 33 MB / 155 MB) — with blocks containing up to ~3,800 events (1.6 MB JSON lines). The data is not corrupt and not gapped: both chunks are valid, contiguous JSON (verified 11.lz4 ends at 1183474774, 12.lz4 starts at 1183474775, no gaps/dupes).

Ingest.runService() sets app.setSocketTimeout(this.getSocketTimeout()), which becomes the Node HTTP server's inactivity timeout. HyperliquidIngest inherits the base default of 120_000 ms. HyperliquidArchive.readRawChunk reads the whole object and lz4.decodes it before streaming, so an oversized chunk produces a long quiet period (measured S3 GET alone ≈ 24 s) plus writer backpressure while 439 MB is streamed. That exceeds 120 s, so the server force-closes the socket every cycle; the writer never receives a complete chunk and the cursor never advances. The failure period of exactly ~125 s (= 120 s timeout + 5 s writer retry pause) pins this to the socket timeout.

The sibling service hyperliquid-replica-cmds-ingest already overrides getSocketTimeout() to 600_000 for the same reason; hyperliquid-fills-ingest — which handles the largest, burstiest chunks — was left on the 120 s default. This PR applies the same override.

Fix

Override getSocketTimeout() in HyperliquidIngest to 600_000, matching hyperliquid-replica-cmds-ingest.

Verification

• Pulled both chunks from s3://hl-mainnet-node-data/node_fills_by_block/hourly/20261014/: 12.lz4 decompresses to 439 MB (vs 155 MB for 11.lz4), contiguous and valid — confirming a volume surge, not corruption.
• Behavior on compiled output, red→green: pre-fix HyperliquidIngest.getSocketTimeout() returns 120000; post-fix returns 600000 (> the base default).
• rush build --to @subsquid/hyperliquid-fills-ingest passes (compiles clean).

Falsification

If, after deploy, the writer still stalls on this chunk with connection closed before message completed at a new fixed period, the bottleneck is not the 120 s timeout (e.g. the writer or R2 upload is slower than 600 s for a single chunk), and the durable fix is to stream-decode chunks in HyperliquidArchive.readRawChunk (avoid buffering the whole file) and/or have the writer commit smaller sub-ranges — noted as a follow-up.