Skip to content

fix(runners): preserve UTF-8 across child output chunks - #469

Open
wcy12378 wants to merge 1 commit into
Tencent:mainfrom
wcy12378:codex/preserve-child-output-utf8
Open

wcy12378 wants to merge 1 commit into
Tencent:mainfrom
wcy12378:codex/preserve-child-output-utf8

Conversation

@wcy12378

Copy link
Copy Markdown

Problem

The DSH process runner and browser-eval process helper decode each stdout/stderr Buffer independently. A UTF-8 character split across pipe chunks is replaced with U+FFFD characters, silently corrupting page text and CLI diagnostics passed to the agent or evaluation oracle. A child that writes the bytes of 中文🙂 separately reproduces this on current main.

Change

Use a separate streaming UTF-8 decoder for each DSH output stream, flush unfinished sequences when collection settles, and use Node's readable-stream UTF-8 decoding in the eval helper. Add regression tests for split multibyte output on stdout/stderr and incomplete trailing sequences. No new dependencies.

Validation

Environment: Windows x64, Node.js 24.15.0.

  • Before the fix, both new split-output tests fail with replacement characters instead of the original text.
  • node --test evals/browser/tests/*.test.mjs: baseline 23 passed; with the patch 25 passed.
  • Focused DSH runner/runner-exit tests, using cached Vitest 5.0.0 and an isolated config: baseline 51 passed / 2 failed (new tests excluded); final 53 passed / the same 2 failed. The existing Windows failures wait 4 seconds while the runner's Windows cancellation grace is 15 seconds (settles after the kill grace when neither exit nor close ever fires, settles killAll and killFor children that never report back).
  • Strict standalone TypeScript 5.8.3 check of src/runner.ts passed; git diff --check passed.
  • The repository's full pinned toolchain, package-wide typecheck, lint, extension tests, and Rust suite were not run successfully. Dependency installation encountered registry connection resets; the focused Vitest run is not the repository's pinned Vitest 4.1.6 configuration.

Related work

Open PRs #404 and #379 also touch the DSH runner, but their changes are confined to rendering command errors; they do not change stream decoding. This patch changes output collection and the eval helper.

AI assistance

OpenAI Codex assisted with source inspection, implementation, regression tests, and the listed validation.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant