Skip to content

fix: tear down brokers by sessionId on SessionEnd to avoid orphaned worktree brokers (#380)#381

Open
hongsu wants to merge 12 commits into
openai:mainfrom
hongsu:fix/broker-session-cleanup
Open

fix: tear down brokers by sessionId on SessionEnd to avoid orphaned worktree brokers (#380)#381
hongsu wants to merge 12 commits into
openai:mainfrom
hongsu:fix/broker-session-cleanup

Conversation

@hongsu

@hongsu hongsu commented Jun 18, 2026

Copy link
Copy Markdown

Summary

Fixes #380.

handleSessionEnd tears down the broker via loadBrokerSession(input.cwd), and broker.json is stored under a cwd-derived hash (resolveStateDir). When a broker was spawned for a different cwd than the session's cwd, the lookup misses, no broker/shutdown is sent, and the broker is orphaned — even on graceful exit (/quit, SIGTERM).

This is structural in a common workflow: a session runs in the main tree (session cwd = repo root) but runs reviews against a git worktree (broker cwd = worktree). Reviewing a worktree's uncommitted working tree requires broker cwd = worktree, so the mismatch is unavoidable, and graceful exit still leaks the broker.

Fix

Decouple broker cleanup from cwd by keying it on the owning session(s):

  • Record session ownership in broker.json: broker creation records the creating session (resolveSessionId reads CODEX_COMPANION_SESSION_ID, which the SessionStart hook already sets), and reusing a ready broker adds the reusing session as a co-owner (sessionIds). A broker shared by several sessions is torn down by the last owner to end, not by whichever session happens to exit first.
  • Record a best-effort session pid per owner (sessionPids; the SessionStart hook exports CODEX_COMPANION_SESSION_PID from its parent pid). Teardown prunes co-owners whose pid no longer exists, so a session that died without SessionEnd (SIGKILL/OOM) cannot keep the broker alive forever. Owners without a recorded pid are assumed alive (legacy behavior).
  • teardownBrokersForSession(sessionId) scans the shared state root (resolveStateRoot()) and, for every broker.json owned by the ending session: removes the owner, prunes dead co-owners, and shuts the broker down when no live owner remains — independent of cwd.
  • handleSessionEnd calls it with input.session_id in addition to the existing cwd-based path (kept as a fast path / fallback for legacy brokers without ownership records). The cwd path only backs off while the broker still has live owners.

Robustness hardening on these paths:

  • Owner updates are serialized with a per-state-file lock (withBrokerStateFileLock, mkdir-based with stale-lock reclaim), so concurrent SessionEnd hooks cannot lose an owner update or leave an already-ended owner behind.
  • sendBrokerShutdown is bounded (1s default, configurable) so one unresponsive endpoint cannot hang SessionEnd; local teardown/unlink still proceeds after the timeout.
  • Broker teardown runs even when job cleanup throws (cleanupSessionJobs failure is captured and rethrown after broker cleanup).
  • isExecutedDirectly() no longer crashes at import time when process.argv[1] does not resolve to a real path.

Scope

In scope: orphaning on graceful exit caused by cwd mismatch (the dominant path), plus not letting a crashed co-owner pin a shared broker once any other owner ends gracefully.

Out of scope (separate concerns): a broker whose every owner died without SessionEnd still needs an idle-timeout / parent-liveness watchdog (see #108). Those are complementary and tracked separately.

Changes

  • scripts/lib/state.mjs — extract resolveStateRoot() (used to scan all workspaces' state without a cwd); resolveStateDir now reuses it (behavior unchanged).
  • scripts/lib/broker-lifecycle.mjsresolveSessionId() / resolveSessionPid(); owner bookkeeping (sessionIds, sessionPids) on create and reuse; teardownBrokersForSession(sessionId, {killProcess, shutdownTimeoutMs}) with dead-owner pruning; withBrokerStateFileLock; bounded sendBrokerShutdown.
  • scripts/session-lifecycle-hook.mjs — export handleSessionEnd; SessionStart also exports CODEX_COMPANION_SESSION_PID; SessionEnd runs the session-keyed teardown before the cwd-based path and only backs off for live owners; job-cleanup failures no longer skip broker teardown. A small isMain guard wraps the top-level main() so importing the module for tests does not trigger the blocking stdin read (CLI hook execution is preserved).

Backward compatibility

  • STATE_VERSION unchanged; the sessionId / sessionIds / sessionPids fields in broker.json are additive.
  • Existing loadBrokerSession / teardownBrokerSession / clearBrokerSession signatures and the cwd-based cleanup path are retained. Legacy broker.json files without ownership records are skipped by the session scan, treated as always-live by the pruning, and still handled by the existing cwd path.

Tests

New tests/broker-lifecycle.test.mjs (node:test), 20 cases covering resolveStateRoot, resolveSessionId / resolveSessionPid precedence, teardownBrokersForSession (cross-cwd teardown, non-matching sessionId left intact, legacy/no-sessionId ignored, empty-sessionId no-op, unresponsive-endpoint shutdown timeout, concurrent SessionEnd race via child processes), shared-broker ownership transfer on reuse, dead co-owner pruning (dead pid → teardown; live pid → retained), handleSessionEnd regression locks (cwd-mismatch teardown, shared-cwd owner removal, teardown despite job-cleanup failure, dead-sole-owner reclamation), plus runtime coverage for the SessionStart env exports and symlinked plugin roots.

node --test tests/*.test.mjs: the new suite is 20/20; no new failures introduced.

🤖 Generated with Claude Code

hongsu and others added 4 commits June 18, 2026 16:07
브로커 생성 시 CODEX_COMPANION_SESSION_ID를 broker.json에 sessionId 필드로 기록.
resolveSessionId(options)는 options.sessionId → options.env → process.env → null 순서로 해소.
기존 세션 재사용 경로(early-return)는 변경 없음.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@hongsu hongsu requested a review from a team June 18, 2026 07:31

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 8aa56c3f48

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread plugins/codex/scripts/session-lifecycle-hook.mjs Outdated
Compare the hook entrypoint and argv path after resolving realpaths so Node's import.meta.url canonicalization does not bypass direct execution when CLAUDE_PLUGIN_ROOT is a symlink. Add a runtime regression test that invokes SessionStart through a symlinked plugin root and verifies the exported session environment.

Constraint: Node resolves imported module URLs to real paths while argv preserves the invoked symlink path.
Rejected: Raw URL pathname comparison | fails for symlinked plugin installations.
Confidence: high
Scope-risk: narrow
Directive: Keep hook direct-execution checks symlink-aware; SessionEnd cleanup depends on main() running from installed hook commands.
Tested: node --test tests/runtime.test.mjs tests/broker-lifecycle.test.mjs; npm test
Not-tested: npm run build fails before this change because generated app-server types do not match current source types.
@hongsu hongsu force-pushed the fix/broker-session-cleanup branch from 817197c to 82a372f Compare June 18, 2026 09:23

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 82a372fc1f

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread plugins/codex/scripts/lib/broker-lifecycle.mjs Outdated

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 55b5892f52

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread plugins/codex/scripts/lib/broker-lifecycle.mjs Outdated
Serialize broker owner updates with a per-state-file lock so concurrent SessionEnd hooks cannot leave an already-ended owner behind. Ensure broker teardown still runs when job cleanup throws, and bound broker shutdown RPC waits so unresponsive endpoints cannot hang SessionEnd. Add regressions for each path.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: c22bb77eab

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread plugins/codex/scripts/lib/broker-lifecycle.mjs Outdated
hongsu and others added 2 commits July 7, 2026 18:42
A co-owner that dies without SessionEnd (SIGKILL, OOM, crash) stays in
sessionIds forever, so the last graceful SessionEnd early-returns and the
broker is orphaned even though every owning session is gone.

Record a best-effort session pid (the SessionStart hook's parent, exported
as CODEX_COMPANION_SESSION_PID) alongside each owner. teardownBrokersForSession
now prunes co-owners whose recorded pid no longer exists before deciding
whether the broker must stay up, and handleSessionEnd's early-return only
respects owners that are still live. Owners without a recorded pid keep the
previous assume-alive behavior, so legacy broker.json files are unaffected.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
fs.realpathSync throws ENOENT when process.argv[1] does not resolve to a
real path (deleted script, loader indirection), which would crash the hook
module at import time. Treat an unresolvable path as not-executed-directly.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 19a722c638

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread plugins/codex/scripts/session-lifecycle-hook.mjs Outdated
The reuse path probed readiness before acquiring the state-file lock and
then fell back to the pre-lock snapshot (loadBrokerSession(cwd) ?? existing).
When the last owner's SessionEnd won the lock first, the fallback resurrected
the just-shut-down broker: the caller got a dead endpoint and broker.json was
rewritten to point at it.

Reuse now trusts only the locked re-read: if broker.json is gone or points at
a different endpoint, the attempt is retried once (a live replacement broker
is reused) and otherwise falls through to spawning a fresh broker. The spawn
path also re-reads the current state instead of tearing down the stale
pre-lock snapshot.

Regression-locked with a deterministic lock-hold test (torn-down broker is
not resurrected; a fresh broker is spawned) plus a live-replacement reuse
test.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: da7a24beb1

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread plugins/codex/scripts/lib/broker-lifecycle.mjs Outdated
…urrency

Revert the session-PID owner-liveness mechanism (recording process.ppid at
SessionStart and pruning owners whose pid is gone). Under the packaged
shell-form SessionStart hook, node's process.ppid is the ephemeral sh that
wraps the command, not the Claude session, so the recorded pid is dead almost
immediately. That made every shared-broker co-owner look dead, so any single
session exit tore down a broker still in use by its siblings — regressing the
shared-ownership guarantee. The dead-co-owner-orphan case it targeted is the
abnormal-exit concern already deferred to the broker idle-timeout (openai#108).

Also fix two concurrency defects surfaced while reviewing that code:

- teardownBrokersForSession aborted the entire state-root scan when any single
  broker.json.lock could not be acquired within the timeout, so a fresh lock
  held by an unrelated/stuck hook left the ending session's own brokers
  uncollected. Lock timeouts are now caught per entry and the scan continues.

- ensureBrokerSession's fresh-broker spawn+persist ran outside the state-file
  lock, so two racing sessions could each spawn a broker and clobber
  broker.json, orphaning one broker with no record to clean it later. Spawn is
  now serialized under the lock (adopt a live broker written by a racing
  session, else spawn), with an unlocked best-effort fallback only when the
  lock cannot be acquired.

- The state-file lock now stamps an owner token so a stale reclaim by another
  waiter cannot have its lock directory deleted by the original holder's
  release path.

Tests: drop the pid-injection tests; add a locked-entry-skip teardown test and
a dead-record fresh-spawn test. Broker suite 19/19; no new failures in the
full suite (the 9 remaining failures are pre-existing environmental cases
present at the merge base).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 60671d2b81

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

return;
}
const remainingOwners = owners.filter((owner) => owner !== sessionId);
if (remainingOwners.length > 0) {

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Prune dead co-owners before preserving brokers

When a shared broker's other owner has already disappeared without running SessionEnd (for example, the Claude session was SIGKILLed), this branch treats that stale owner as live, rewrites broker.json to only the dead session, and returns without shutting the broker down. Since no future hook will run for that owner, the worktree broker remains orphaned indefinitely; fresh evidence after the PID-liveness fix is that this remaining-owner branch now performs no liveness check before preserving the broker.

Useful? React with 👍 / 👎.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

SessionEnd cleanup fails when review cwd ≠ session cwd — broker.json looked up by cwd-hash, leaving orphan brokers even on graceful /quit

1 participant