fix(menubar): run CLI exit-wait and timeout off the cooperative pool#426
Merged
Conversation
The menubar wedged on "Loading Today…" for hours after an idle period. Root cause: DataClient.runCLI called the blocking process.waitUntilExit() from an async function on Swift's cooperative thread pool. On a 16-core machine, 16 concurrent slow `codeburn` subprocesses pinned all 16 cooperative threads inside waitUntilExit; the 45s timeout — itself a Task on that same pool — could then never be scheduled to kill them, so the deadlock was permanent. Confirmed via sample: 16/16 cooperative threads parked in waitUntilExit. PR #412 (AppStore inFlightKeys bookkeeping) was a layer above the OS-thread deadlock and could not fix it. Move both blocking points off the cooperative pool: bridge waitUntilExit through a global (overcommit) queue via a continuation, and drive the timeout from a DispatchSource on a global queue so it fires even when the pool is saturated. Extract runProcess for testability; add a concurrency + timeout smoke test and an output/exit-code test.
iamtoruk
added a commit
that referenced
this pull request
Jun 9, 2026
…read; cap CLI spawns The #426 fix moved waitUntilExit and its timeout onto the same global(qos:.utility) queue. Under sustained load every utility worker blocked in waitUntilExit, so the timeout could never be scheduled to kill them and the menubar wedged on Loading forever (confirmed via sample after ~a week of soak). Await process.terminationHandler (fires on a Foundation queue, blocks no worker) so the timeout always has a free thread. Add an actor-based async semaphore capping concurrent CLI spawns at 6.
iamtoruk
added a commit
that referenced
this pull request
Jun 9, 2026
…read; cap CLI spawns (#462) The #426 fix moved waitUntilExit and its timeout onto the same global(qos:.utility) queue. Under sustained load every utility worker blocked in waitUntilExit, so the timeout could never be scheduled to kill them and the menubar wedged on Loading forever (confirmed via sample after ~a week of soak). Await process.terminationHandler (fires on a Foundation queue, blocks no worker) so the timeout always has a free thread. Add an actor-based async semaphore capping concurrent CLI spawns at 6.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The menubar wedged on "Loading Today…" for hours after an idle period. This moves the two blocking points in
DataClient.runCLIoff Swift's cooperative thread pool so a saturated pool can no longer deadlock the CLI calls or their timeout.Root cause
DataClient.runCLIcalled the blockingprocess.waitUntilExit()from an async function on the cooperative thread pool. On a 16-core machine, 16 concurrent slowcodeburnsubprocesses pinned all 16 cooperative threads insidewaitUntilExit; the 45s timeout — itself aTaskon that same pool — could then never be scheduled to kill them, so the deadlock was permanent. Confirmed viasample: 16/16 cooperative threads parked inwaitUntilExit.PR #412 (AppStore
inFlightKeysbookkeeping) sat a layer above this OS-thread deadlock and could not fix it.Fix
waitUntilExitthrough a global (overcommit) queue via a continuation.DispatchSourceon a global queue so it fires even when the cooperative pool is saturated.runProcessfor testability.Testing
DataClientProcessTests: concurrency + timeout smoke test, output/exit-code test.