fix: keep only the tail of git's stderr in every git runner's error by ThomasK33 · Pull Request #776 · coder/openstorage

ThomasK33 · 2026-11-08T00:56:48Z

Why

#759 bounded git's stderr in gitread.Run and the write path's git helpers with gitlimit.TailBuffer, the last 64 KiB. The other git runners still kept all of git's stderr and quoted it in their errors, and those reach the log:

githttp: runGitCtx, the for-each-ref stream (localRefSeq), update-ref --stdin and the commit-graph write.
jobs: runGitStdin.
sync: runGitTo.
lfsscan: the three children of ReachableLFSOIDs (rev-list, cat-file --batch-check, cat-file --batch).
simtest: simGit, whose errors reach the test log.
The write path's wcBlobSink (hash-object -w --stdin), whose finish error the commit-pack handler logs.

A git that prints 1 MiB to stderr gave an error of about 1 MB in each.

What changes

Each of them writes git's stderr into gitlimit.TailBuffer. git prints its fatal message last.
update-ref --stdin used CombinedOutput for its error text. It prints nothing on stdout, so it now reads only stderr, through refactor(gitlimit): hold a shared lease in every git child of a cache repository #768's RunIn.
The repack job's streamShowIndex (git show-index) keeps its stderr in gitlimit.TailBuffer too: its error reaches the repack job's log.
The pre-receive hook's verifyCommit (git verify-commit) kept all of the verifier's output in memory. The verifier prints its reason first, so it now keeps the head in the new gitlimit.HeadBuffer (the first 64 KiB), and the hook-errors log line names how many bytes it dropped. The pusher still gets the fixed reason.
gitlimit.TailBuffer held a backing array as large as the largest single write (64 MiB for one Write of 64 MiB). It now copies only the last 64 KiB of a large write, so it holds about 128 KiB at most. A cut can land inside a UTF-8 sequence, so a tail that was cut now starts at the next character.
gitlimit.CombinedOutput has no caller left and is removed. refactor(gitlimit): hold a shared lease in every git child of a cache repository #768's CombinedOutputIn stays: its lease audit names it.
One exception keeps all of git's stderr: applyPatch (diff-commit's git apply), as fix(gitlimit): keep only the tail of git's stderr in git errors #759 chose, because its status comes from every line of stderr. Its plain error (a cancelled request or a killed git) quotes only the last 64 KiB. Its status answer (wcApplyRejection: 409, 400 or 500) quotes the first 4096 bytes. The buffer grows with the patch, to about the patch's size or a little more: git apply prints a line per file or hunk that fails, and diff-commit caps each chunk at 4 MiB but not the whole patch. With git 2.55.0, a 1,735,560-byte patch of 20,000 files gave 848,890 bytes of stderr, and one failing one-hunk diff repeated 20,000 times (1,060,000 bytes) gave 1,120,000 bytes. A follow-up PR (fix(api): cap diff-commit's patch, and read git apply's stderr as it streams #778) caps the patch and reads the stderr as it streams.
Their command lines stay as they are: fix(api): clip long client values in git error names and log lines #757 found their arguments server-built.
githttp's runGitCtx, localRefSeq and updateRefStdin, the hook's verifyCommit, and sync's runGitTo, and their imports, are covered by the verify-lane-schedule fingerprints, so this PR re-records them and moves the SKILL.md sync commit. The model does not change.
docs/how-to/deploy-production.md states the bound that fix(api): clip long client values in git error names and log lines #757 put on the method and path fields of the API's log lines. The sentence moved here from fix(api): cap the request path and method in log records #761, which fix(api): clip long client values in git error names and log lines #757 superseded.

Verification

Each runner gets a test with a git that prints 1 MiB and then fatal: the end to stderr (each lfsscan child on its own, through a git that fails only that child). With the old code (go test -overlay):

git_stderr_tail_test.go:42: runGitCtx: the error is 1048617 bytes, want git's stderr cut to its last 64 KiB
git_stderr_tail_test.go:43: updateRefStdin: the error is 1048625 bytes, want git's stderr cut to its last 64 KiB
git_stderr_tail_test.go:53: localRefSeq: the error is 1048623 bytes, want git's stderr cut to its last 64 KiB
git_stderr_tail_test.go:59: runCommitGraphAttempt: the error is 1048629 bytes, want git's stderr cut to its last 64 KiB
git_stderr_tail_test.go:42: runGitStdin: the error is 1048617 bytes, want git's stderr cut to its last 64 KiB
git_stderr_tail_test.go:43: runGitTo: the error is 1048606 bytes, want git's stderr cut to its last 64 KiB
gitenv_test.go:24: simGit: the error is 1048617 bytes, want git's stderr cut to its last 64 KiB
git_stderr_test.go:43: wcBlobSink: the error is 1048618 bytes, want git's stderr cut to its last 64 KiB
git_stderr_tail_test.go:48: streamShowIndex: the error is 1048621 bytes, want git's stderr cut to its last 64 KiB
git_stderr_tail_test.go:83: the error lost the head of the output or the dropped count: git verify-commit aaaa...: exit status 128 (output: "gpg: the reason; eeee...

TestVerifyCommitKeepsTheHeadOfItsOutput uses a git that writes gpg: the reason and then 64 MiB. Against main's TailBuffer:

errtext_test.go:41: writes of 67108864 bytes: the buffer holds 67108864 bytes, want at most 131072
errtext_test.go:59: tail starts "\xac\xe2\x82" and holds 65536 bytes, want whole runes

A TailBuffer that keeps everything and cuts only in String() fails the first of them, and one without the character rule fails the second.

Each lfsscan child with a plain buffer fails its own case: 1,048,615 bytes for rev-list, 1,048,629 for --batch-check and 1,048,623 for --batch. The lfsscan test passed 20 of 20 runs with -race.

Lanes at b40b177, each test lane under the host's shared heavy-validation lock:

go build ./..., go vet ./..., golangci-lint (0 issues) and scripts/check-doc-links.py (0 broken links): pass.
go test -race of internal/api, internal/gitlimit, internal/jobs, internal/lfsscan, internal/simtest, internal/sync and cmd/openstorage, and of internal/githttp on its own, on PostgreSQL, local and S3-WAL: ok.
The local run of the seven packages took three tries. Another process on the host ended the first one with SIGTERM (exit 143). The second one failed 10 internal/sync tests at setup, for example local_schedule_test.go:921: heldport: listen 127.0.0.1:0: listen tcp4 127.0.0.1:0: bind: address already in use: the host had run out of local ports (11,621 sockets in TIME_WAIT just after it ended). The third one passed.

Round 2 and 3 heads, each test lane under the heavy-validation lock: after the merge of main at f3fc471 (#768), check.sh --record (TLC, RESULT: PASS), both --fingerprints-only checks, build, vet, golangci-lint (0 issues) and the link check pass on 2f32d8e, and so do go test -race of internal/gitlimit/... and the touched tests of internal/jobs and internal/githttp (StderrTail|VerifyCommit|VerifySig|Repack|ShowIndex) on PostgreSQL, SQLite and S3-WAL. On the round-3 head, internal/gitlimit passes with -race, and the stderr tests of githttp, jobs, lfsscan and api pass on SQLite.

After the merge of main at 536ab6a, at 5b5a29e: check.sh --record (TLC, RESULT: PASS), the --fingerprints-only check of verify-lane-schedule and verify-orphan-sweep, build, vet, lint and the link check pass. I did not run the test lanes again locally on this head. CI runs them.

Size

490 changed lines: 122 of production Go, 338 of tests, 16 of docs (the gitlimit README included) and 14 of the verify-lane-schedule record.

Generated with xum • Model: anthropic:claude-opus-5-5 • Thinking: xhigh

The git runners outside gitread and the write path kept all of git's stderr and quoted it in their errors, which reach the log: githttp's runGitCtx, its for-each-ref stream, update-ref --stdin and the commit-graph write, jobs' runGitStdin, sync's runGitTo, the three lfsscan scan children and simtest's simGit. Each now writes git's stderr into gitlimit.TailBuffer (the last 64 KiB). update-ref --stdin used CombinedOutput for its error text: it prints nothing on stdout, so it now reads only stderr. Their command lines stay as they are: Each runner gets a test with a git that prints 1 MiB to stderr (each lfsscan child on its own). With the old code the errors were 1,048,606 to 1,048,629 bytes. deploy-production.md states the bound that #757 put on the method and path fields of the API's log lines (moved from #761, which #757 superseded). Change-Id: I32251053afd0f88122bfae3715c94486bc8dcd02 Signed-off-by: Thomas Kosiewski <tk@coder.com>

wcBlobSink kept all of git hash-object's stderr in a bytes.Buffer and finish() quoted it in its error, which the commit-pack handler logs. It now writes stderr into gitlimit.TailBuffer. TestWriteGitErrorsKeepStderrTail gets the sink: with the old buffer its error was 1,048,618 bytes. Change-Id: I32a66facd0bd0bf547a8bbfdad78689aa9c6e394 Signed-off-by: Thomas Kosiewski <tk@coder.com>

Change-Id: I857bd6d7ae9c39cc01b3671d77ba317dfda2e817 Signed-off-by: Thomas Kosiewski <tk@coder.com>

…err tail change Change-Id: I42a000fc02f487178ffc115c099fb987eb77e43f Signed-off-by: Thomas Kosiewski <tk@coder.com>

Change-Id: I40ff8ea02948b8dc6e75cfe1cb75d65e262bf90a Signed-off-by: Thomas Kosiewski <tk@coder.com>

… main Change-Id: Iade359fede2b7d4e21f9fb609d010f3696cb87b1 Signed-off-by: Thomas Kosiewski <tk@coder.com>

ThomasK33 · 2026-11-08T00:56:57Z

@codex review

ThomasK33 · 2026-11-08T00:57:04Z

@codex security review

chatgpt-codex-connector · 2026-11-08T00:57:10Z

Codex Review Summary: ✅ Completed

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	✅ Completed 2026-11-08T03:36:35.587558Z	`e1f53d3`	Manual request
🔒 Security Review	✅ Completed 2026-11-08T03:36:47.819464Z	`e1f53d3`	Manual request

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

chatgpt-codex-connector · 2026-11-08T01:00:16Z

Codex Review: Didn't find any major issues. Nice work!

Reviewed commit: 5b5a29eb21

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

chatgpt-codex-connector · 2026-11-08T01:01:20Z

🛡️ Codex Security Review

Security review completed. No security issues were found in this pull request.

Reviewed commit: 5b5a29eb21

View security finding report

Only the user who started this review can view the report in Codex.

ℹ️ About Codex security reviews in GitHub

This is an experimental Codex feature. Security reviews are triggered when:

You comment "@codex security review"
A regular code review gets triggered (for example, "@codex review" or when a PR is opened), and you’re opted in so security review runs alongside code review

Once complete, Codex will leave suggestions, or a comment if no findings are found.

quoted it in an error that reaches the repack log. It now keeps the tail in gitlimit.TailBuffer. verifyCommit kept all of git verify-commit's output in memory. The verifier's reason comes first, so it now keeps the head in gitlimit.HeadBuffer (the first 64 KiB) and names the count of dropped bytes. gitlimit.CombinedOutput has no caller left and goes. Change-Id: Ib945877dead7e2dc9526243c4980f038f558addf Signed-off-by: Thomas Kosiewski <tk@coder.com>

…g verify-commit's output Change-Id: I01b8acd8a9864862cc8b68accc4815e5ab79ccb9 Signed-off-by: Thomas Kosiewski <tk@coder.com>

ThomasK33 · 2026-11-08T02:18:51Z

@codex review

ThomasK33 · 2026-11-08T02:18:57Z

@codex security review

Change-Id: Ieb677c1826d164c3be3b6167f44dff07f936818c Signed-off-by: Thomas Kosiewski <tk@coder.com>

chatgpt-codex-connector · 2026-11-08T02:21:42Z

Codex Review: Something went wrong. Try again later by commenting “@codex review”.

Unknown error

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

chatgpt-codex-connector · 2026-11-08T02:22:52Z

🛡️ Codex Security Review

Security review completed. No security issues were found in this pull request.

Reviewed commit: aee288cfcc

View security finding report

Only the user who started this review can view the report in Codex.

ℹ️ About Codex security reviews in GitHub

This is an experimental Codex feature. Security reviews are triggered when:

You comment "@codex security review"
A regular code review gets triggered (for example, "@codex review" or when a PR is opened), and you’re opted in so security review runs alongside code review

Once complete, Codex will leave suggestions, or a comment if no findings are found.

… main again Change-Id: I8388359579c3f2624bb7db4d8d1fb9b2d69ecf59 Signed-off-by: Thomas Kosiewski <tk@coder.com>

…at a character TailBuffer that kept everything passed them. One Write of 64 MiB did keep a 64 MiB backing array. Write now copies only the last 64 KiB of a large write. A cut can land inside a UTF-8 sequence, so String drops the continuation bytes at the start of a tail that was cut. Change-Id: Id69fe6e000505a4f9c67292bfaa0434c2b3b7208 Signed-off-by: Thomas Kosiewski <tk@coder.com>

ThomasK33 · 2026-11-08T03:32:43Z

@codex review

ThomasK33 · 2026-11-08T03:32:50Z

@codex security review

chatgpt-codex-connector · 2026-11-08T03:36:34Z

Codex Review: Didn't find any major issues. Another round soon, please!

Reviewed commit: e1f53d3a76

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

chatgpt-codex-connector · 2026-11-08T03:36:46Z

🛡️ Codex Security Review

Security review completed. No security issues were found in this pull request.

Reviewed commit: e1f53d3a76

View security finding report

Only the user who started this review can view the report in Codex.

ℹ️ About Codex security reviews in GitHub

This is an experimental Codex feature. Security reviews are triggered when:

You comment "@codex security review"
A regular code review gets triggered (for example, "@codex review" or when a PR is opened), and you’re opted in so security review runs alongside code review

Once complete, Codex will leave suggestions, or a comment if no findings are found.

ThomasK33 added 6 commits November 7, 2026 15:44

Merge remote-tracking branch 'origin/main' into stderr-tail-git-runners

c30574e

Change-Id: I857bd6d7ae9c39cc01b3671d77ba317dfda2e817 Signed-off-by: Thomas Kosiewski <tk@coder.com>

chore(verify-lane-schedule): re-record the fingerprints after the std…

b40b177

…err tail change Change-Id: I42a000fc02f487178ffc115c099fb987eb77e43f Signed-off-by: Thomas Kosiewski <tk@coder.com>

Merge remote-tracking branch 'origin/main' into stderr-tail-git-runners

1dc7801

Change-Id: I40ff8ea02948b8dc6e75cfe1cb75d65e262bf90a Signed-off-by: Thomas Kosiewski <tk@coder.com>

chore(verify-lane-schedule): re-record the fingerprints after merging…

5b5a29e

… main Change-Id: Iade359fede2b7d4e21f9fb609d010f3696cb87b1 Signed-off-by: Thomas Kosiewski <tk@coder.com>

ThomasK33 added 2 commits November 8, 2026 01:45

chore(verify-lane-schedule): re-record the fingerprints after boundin…

aee288c

…g verify-commit's output Change-Id: I01b8acd8a9864862cc8b68accc4815e5ab79ccb9 Signed-off-by: Thomas Kosiewski <tk@coder.com>

Merge remote-tracking branch 'origin/main' into stderr-tail-git-runners

ed3492f

Change-Id: Ieb677c1826d164c3be3b6167f44dff07f936818c Signed-off-by: Thomas Kosiewski <tk@coder.com>

ThomasK33 added 2 commits November 8, 2026 03:06

chore(verify-lane-schedule): re-record the fingerprints after merging…

2f32d8e

… main again Change-Id: I8388359579c3f2624bb7db4d8d1fb9b2d69ecf59 Signed-off-by: Thomas Kosiewski <tk@coder.com>

ThomasK33 mentioned this pull request Nov 8, 2026

fix(api): cap diff-commit's patch, and read git apply's stderr as it streams #778

Merged

ThomasK33 added this pull request to the merge queue Nov 8, 2026

Merged via the queue into main with commit 50df23a Nov 8, 2026
11 of 12 checks passed

ThomasK33 deleted the stderr-tail-git-runners branch November 8, 2026 06:03

github-actions Bot mentioned this pull request Nov 8, 2026

chore(main): release 0.20.0 #653

Open

Conversation

ThomasK33 commented Nov 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Why

What changes

Verification

Size

Uh oh!

ThomasK33 commented Nov 8, 2026

Uh oh!

ThomasK33 commented Nov 8, 2026

Uh oh!

chatgpt-codex-connector Bot commented Nov 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

chatgpt-codex-connector Bot commented Nov 8, 2026

Uh oh!

chatgpt-codex-connector Bot commented Nov 8, 2026

🛡️ Codex Security Review

Uh oh!

ThomasK33 commented Nov 8, 2026

Uh oh!

ThomasK33 commented Nov 8, 2026

Uh oh!

chatgpt-codex-connector Bot commented Nov 8, 2026

Uh oh!

chatgpt-codex-connector Bot commented Nov 8, 2026

🛡️ Codex Security Review

Uh oh!

ThomasK33 commented Nov 8, 2026

Uh oh!

ThomasK33 commented Nov 8, 2026

Uh oh!

chatgpt-codex-connector Bot commented Nov 8, 2026

Uh oh!

chatgpt-codex-connector Bot commented Nov 8, 2026

🛡️ Codex Security Review

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

ThomasK33 commented Nov 8, 2026 •

edited

Loading

chatgpt-codex-connector Bot commented Nov 8, 2026 •

edited

Loading