[GUI] Agent turn is ended after 4 minutes while the local engine is still reading a long prompt (the watchdog ignores engine progress) #3623

Area

[GUI] (the desktop app)

Desktop app, Terminal or source?

Desktop app (Windows, Linux or Mac)

What happened

In an agent (coding assistant) chat with a local model, the turn was ended with:

The assistant stopped answering for 4 minutes, so I ended this turn. Anything it already finished is kept. Send your message again to go on.

The model had not stopped. The engine was reading a 59,091-token prompt. Only 6,154 tokens came from the prefix cache, and it had read 52,932 of the remaining 52,937 at 208 tok/s when the turn was cancelled at 255 s:

strata serve: prompt 59091 tokens = 6154 reused + 52932 of 52937 read in 254492 ms (208.0 tok/s), 0 generated in 0 ms (0.0 tok/s), drafts accepted 0 of 0, 4 checkpoints (cancelled)

The request just before it (85,480 tokens: 68,036 reused, 17,444 read in 95 s) finished normally. The engine was healthy. The cache miss is what made this read take longer than 4 minutes.

[strata] reading the prompt: 59,091 tokens, 10 s so far
[strata] reading the prompt: 12,298 of 59,091 tokens, 28 s so far
...
[strata] reading the prompt: 55,306 of 59,091 tokens, 243 s so far
[strata] done: 0 tokens in 255 s (0.0 tok/s) (cancel, cancel=True)

Expected: while the engine is making progress on the prompt, the turn keeps waiting. It should only be ended when neither the Harness nor the engine has moved for the stall time.

Why it happens (from reading the code)

Two timers can end a healthy request, and neither sees the engine's keep-alives:

1. The agent watchdog hears only the Harness. In apps/host/server/agent/agent-bridge.mjs, turn.watchdog ends the turn when Date.now() - turn.lastActivity >= stallMs. stallMs is 4 min, and there is no setting or environment variable for it. turn.lastActivity is updated only for Harness items (handle, fold). While the engine reads a prompt it sends only : keep-alive SSE comments, which the OpenAI client drops. So the Harness stays silent, and any prompt read longer than 4 minutes ends the turn.

   serve already handles this case: engine_silence_s is extended by prompt-progress lines (serve engine deadlocks in SleepConditionVariableSRW during a long streaming reply; stall watchdog never fires #481), and STRATA_ENGINE_STALL_S only ends an engine that also uses no CPU, disk or GPU (serve: an untimed read of the engine/vision pipe can hold the request turn forever (intermittent hang; the #481 lost-step class) #1317). The host also already polls the engine's progress every 2 s (/metrics → live.state, live.prompt_read, live.generated in native-control.mjs). The watchdog just does not use it.

2. The Harness's LLM client has the same blind spot. @deepseek-ai/dsh-llm-pi-ai drops a request after streamIdleTimeoutMs (default 300000) without a stream event. The error is pi-ai stream idle timeout after 300000ms, code TIMEOUT. openai/core/streaming.js drops lines that start with :, so neither the queue beats nor the prompt-progress comments reset that timer. apps/host/server/agent/profile.mjs sets no value for local providers. With the defaults the watchdog fires first, so fixing (1) alone only moves the cut-off from 4 to 5 minutes. That is why both are in one report. It is the keep-alive-as-comment problem of serve: keep-alive is an SSE comment, so clients that drop comments still hit their stream idle timeout and retry #1951 / PR serve: the keep-alive while a prompt is read is a real SSE frame, not a comment #1953 (closed, not merged), but for Strata's own agent.

   In a 3.5 h run with the timeout raised to 15 min (patch below), the two helpers' sessions show 16 llm/retry events, all with pi-ai stream idle timeout after 900000ms (code TIMEOUT). Seven of the gaps between one retry and the next are 900–902 s, so the retried request timed out again after exactly 15 min. With one slot, a helper's request first waits behind the other agents' turns and then reads its own prompt, so it hears nothing for more than 15 min. For 15 of the 16 retries, the engine log shows the dropped request as (cancelled) with 0 tokens generated, 153 to 752 s into the read. Each was then sent again and read from the start, which took about 1.8 h of the 3.5 h of engine time. A fixed timeout does not fit this case, because the wait grows with the number of agents that share a slot.

Local patch I am running (works for me; not proposed as the fix)

• engineProgress(instanceId), passed from index.mjs, returns a short fingerprint of the engine's live metrics, or null when the engine is idle or its metrics are unavailable.
• For local routes, the watchdog counts a changed fingerprint as activity. Because the fingerprint covers the whole engine, it counts only for engineGraceMs (30 min) after the Harness was last heard. stallMs stays at 4 min, so an idle or stuck engine still ends the turn as before. A lost turn ends at about 34 min, and API routes are unchanged.
• Local providers get streamIdleTimeoutMs: 60 * 60_000 as a backstop. 15 min was not enough once helpers shared the slot. A hung engine is still ended by serve, and a lost turn by the watchdog cap above.
• A real fix would let pi-ai count the engine's keep-alives (or its return_progress chunks) as activity, and leave hang detection to serve (serve engine deadlocks in SleepConditionVariableSRW during a long streaming reply; stall watchdog never fires #481, serve: an untimed read of the engine/vision pipe can hold the request turn forever (intermittent hang; the #481 lost-step class) #1317).

diff --git a/apps/host/server/agent/agent-bridge.mjs b/apps/host/server/agent/agent-bridge.mjs
--- a/apps/host/server/agent/agent-bridge.mjs
+++ b/apps/host/server/agent/agent-bridge.mjs
@@ export function createAgentBridge(...) @@
-export function createAgentBridge({ ..., expertSampling = () => null, stallMs = 4 * 60_000, stallToolMs = 20 * 60_000, ... }) {
+export function createAgentBridge({ ..., expertSampling = () => null, engineProgress = () => null, engineGraceMs = 30 * 60_000, stallMs = 4 * 60_000, stallToolMs = 20 * 60_000, ... }) {
@@ turn.watchdog = setInterval(() => { ... @@
       const limit = toolRunning ? stallToolMs : stallMs;
+      // A local engine reading a long prompt sends the Harness nothing for minutes. Its own progress
+      // (engineProgress: the host's /metrics poll) counts as activity, so only an engine that is idle or stuck lets the turn stall.
+      // It counts for at most engineGraceMs after the Harness was last heard: progress for another chat cannot keep a lost turn forever.
+      if (route.kind === 'local') {
+        if (turn.lastActivity !== turn.engineBump) turn.heard = turn.lastActivity;   // the Harness moved it since the engine last did
+        let mark = null;
+        try { mark = engineProgress(route.instanceId); } catch { /* no progress known: judged by the Harness alone */ }
+        if (mark != null && mark !== turn.engineMark && Date.now() - turn.heard < engineGraceMs) {
+          turn.engineMark = mark; turn.lastActivity = turn.engineBump = Date.now(); return;
+        }
+      }
       if (Date.now() - turn.lastActivity < limit) return;
diff --git a/apps/host/server/index.mjs b/apps/host/server/index.mjs
--- a/apps/host/server/index.mjs
+++ b/apps/host/server/index.mjs
@@ const agent = createAgentBridge({ @@
   expertSampling: () => samplingFromExpert(store.get().expert),
+  // How far a local engine has got (the 2 s /metrics poll), for the agent's stall watchdog. It moves while
+  // a long prompt is read, when the agent hears nothing; null when the engine is idle or says nothing.
+  engineProgress: instanceId => {
+    const live = native.readyInstances().find(i => i.id === instanceId)?.metrics?.live;
+    if (!live || live.state === 'idle' || live.state === 'unloaded') return null;
+    const slots = Array.isArray(live.slots) ? live.slots.map(s => `${s?.state}:${s?.prompt_read ?? ''}:${s?.generated ?? ''}`).join(',') : '';
+    return `${live.state}|${live.prompt_read ?? ''}|${live.generated ?? ''}|${slots}`;
+  },
diff --git a/apps/host/server/agent/profile.mjs b/apps/host/server/agent/profile.mjs
--- a/apps/host/server/agent/profile.mjs
+++ b/apps/host/server/agent/profile.mjs
@@ local.forEach((inst, index) => { ... providers[route] = { @@
       baseURL: inst.baseUrl,
       apiKeyEnv: keyEnv,
+      // A long prompt read sends only SSE comments, which do not reset the 300 s default idle timer.
+      // 60 min: with helpers taking turns on one slot, a request waits for the others and then reads its own prompt (15 min was
+      // exceeded 16 times in one run); a hung engine is ended by serve itself, and a lost turn by the bridge's watchdog.
+      streamIdleTimeoutMs: 60 * 60_000,

Validation (the patch applied to the installed 0.2.1 files on this PC)

Passed:

• node --check with Strata's bundled Node (v24.12.0) on host/server/agent/agent-bridge.mjs, host/server/index.mjs and host/server/agent/profile.mjs: exit 0.
• buildProfile() from the patched profile.mjs, called from a small script with one local instance: streamIdleTimeoutMs: 3600000 on strata-local and on all four strata-local~sN routes.
• A script that takes the patched watchdog block and engineProgress from the installed files and runs them on a simulated clock: 11 of 11 cases pass (list below).
• In the app, the 3.5 h agent run above (one parent and two helpers on this one-slot engine, watchdog patch with a 15 min streamIdleTimeoutMs): 23 prompt reads took 240 s or more, 11 of them finished normally (the longest 483 s), and the watchdog did not end the turn once.
• In the app, the next run with 60 min (2.7 h of engine time, 52 requests): 6 prompt reads took 240 s or more and all finished, with 0 cancelled reads and 0 llm/retry. I turned on the engine's conversation cache (--conversation-cache-mib 4096) at the same time, which also shortens the waits.

Failed:

• 15 min for streamIdleTimeoutMs was not enough: 16 llm/retry in the 3.5 h run, as described in (2).

Not run:

• Strata 0.2.2 / engine 0.1.47, Linux and macOS.
• apps/host/tests/agent-watchdog.test.mjs (it needs a built Harness in STRATA_HARNESS_DIR).
• API-model routes in the app (covered only by the simulation).
• The diff against main. It was made against the installed 0.2.1 files.

Smallest steps or command that reproduce it

Seen once. I have not re-run it on purpose yet.

1. Desktop app, agent chat with the local model below, in a long session with many tool calls (the prompt had grown to ~59k tokens).
2. Branch the chat and send a message in the branch. The prefix cache matched 6,154 tokens, so 52,937 tokens had to be read again at ~208 tok/s (about 254 s).
3. At 255 s the turn ended with the message above. The engine log shows the read still progressing until the cancel.

The trigger (a long re-read after a cache miss) looks related to #3493. The failure itself (the agent watchdog) is separate.

Reproduction status

Reproduced locally

Strata version

Strata 0.2.1 (desktop app) · engine 0.1.46.1 (windows-x86_64-hip-rdna3) · DeepSeek Harness (@deepseek-ai/dsh) 0.2.1-alpha.3. I have not run 0.2.2. As of 2026-10-22, main still has stallMs = 4 * 60_000 with turn.lastActivity set only from Harness items, and profile.mjs sets no streamIdleTimeoutMs.

Graphics card, memory and system

AMD Ryzen AI MAX+ 395 with Radeon 8060S (gfx1151, HIP). The GPU is integrated: Windows reports 512 MiB of dedicated GPU memory (the BIOS carve-out), and the engine works from the 128 GB of unified memory (127.1 GiB visible to Windows). Windows 11 (build 26300).

Model, size and settings

Qwen3.8-Flash-Next UD-Q4_K_XL (Unsloth) · 131072 context (--kv int8 --kv-resident 32768) · pictures on · thinking Max (budget 24576 of max_tokens 32768) · parallel 1

Logs

The excerpts are above. Related: #481, #1317, #1619, #1951, PR #1953, #3493.

Investigated and drafted with Claude Code (code reading, log analysis and writing). The numbers and log lines come from the logs on my machine and were cross-checked against them.