Skip to content

fix(extension): reduce idle overlay and Accessibility overhead - #408

Open
NianJiuZst wants to merge 7 commits into
Tencent:mainfrom
NianJiuZst:fix/391-idle-overlay-ax-cost
Open

NianJiuZst wants to merge 7 commits into
Tencent:mainfrom
NianJiuZst:fix/391-idle-overlay-ax-cost

Conversation

@NianJiuZst

@NianJiuZst NianJiuZst commented Oct 8, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Fixes #391.

An agent session can keep doing browser work between commands: the control overlay continuously animates a viewport-sized inset shadow, and observation enables Chrome's Accessibility domain without disabling it afterward. This PR makes the control glow settle after two opacity pulses and releases Accessibility on the targets whose capture reads have finished.

Problem and resulting behavior

Area Before After
Control glow An infinite box-shadow animation runs for the entire time the overlay is visible. Two static shadow layers are painted; only the brighter layer's opacity animates, for two 3-second cycles, before the resting glow remains.
Reduced motion The breathing animation ignores the user's motion preference. prefers-reduced-motion: reduce disables the pulse and leaves the static control glow.
Observation Each target receives Accessibility.enable, but completed captures leave the domain enabled. Successfully enabled targets receive Accessibility.disable after all required AX, document-identity and geometry reads.
Capture interruption There is no capture-scoped Accessibility release. Failure or cancellation triggers release for the targets already enabled, while attachment and pending-read checks protect cleanup dispatch.

Implementation

Control overlay

ControlOverlay.tsx keeps the resting inset shadow on the existing overlay and adds a brighter child layer. The bsk-breathe keyframes now contain only opacity changes, with a finite iteration count of two. The child has a resting opacity of zero, and a reduced-motion media query disables its animation. The control indicator, interruption button, and input-blocking behavior remain available.

Accessibility lifetime

capture-coordinator.ts records each successfully enabled target together with its debugger attachment ID. Release uses the capture's existing worker helper, with at most four cleanup requests in flight, and is awaited before the capture completes.

  • Release runs in an outer finally around the entire capture, on success, failure, or cancellation.
  • Cleanup uses sendGuarded with the original attachment ID in production, so an old capture cannot reconnect or send cleanup to a replacement attachment.
  • A target whose read times out is removed from the release set because Chrome can still be running it. A pending root read excludes all targets on that tab; an isolated child timeout excludes only its target.
  • command-deadline.ts guards Accessibility.disable against earlier pending reads and gives cleanup itself a 1-second deadline. A timed-out disable retains its target gate until Chrome replies or the debugger detaches; late replies cannot clear a replacement attachment.
  • Rejected cleanup requests are treated as best-effort cleanup and do not replace the capture result or its original error.

Observation refs retain their backend DOM node IDs and document/attachment identity; the implementation does not depend on Accessibility node IDs surviving between captures. The changelog records both user-visible changes.

Regression coverage

Eight new cases extend three existing test files:

  • ControlOverlay.test.tsx (2): static shadows, opacity-only keyframes, exactly two pulses, no infinite animation, and the reduced-motion override.
  • capture-coordinator.test.ts (5): release for root and child targets after their capture reads; release after identity/geometry work; dispatch on the original attachment; no cleanup on a replacement attachment; skipping a timed-out child while releasing the root; and cleanup during cancellation. Some cases assert multiple related behaviors.
  • command-deadline.test.ts (1): a pending child-session AX read blocks its own cleanup without blocking root-target cleanup.

Historical validation — 2026-10-20

Merged upstream main e8777ab71bb851196895515b53eea2481462dd57; tested revision d305f8de8867bc42081d23c765502ab2291ce44b.

  • Extension suite: 2,683 passed, 244 skipped. The three directly affected test files pass 82 tests.
  • Extension TypeScript, changed-file Biome, production build and diff checks passed.
  • Two additional out-of-tree capture regressions cover completion and cancellation with a stuck child cleanup. Both settle within the cleanup deadline; the root remains usable, the affected child stays gated, and its late response restores access.
  • Real Chrome 153: after AX reads, an intentionally blocked renderer kept Accessibility.disable pending. The production read gate returned its timeout after 1,002 ms, rejected another read while pending, and recovered after the original command settled following execution termination.
  • Independent full-PR P0–P2 review passed. New proof scripts/logs remain outside the repository.

Limits and performance evidence

  • Root capture regressions run against real Chrome and the production capture/read-gate implementation; child loss, cancellation, and replacement-gate cases also use simulated CDP. CPU/GPU profiling was not rerun; skipped tests are not counted as passes.
  • The measurements in issue #391 belong to the reporter's combined overlay/Accessibility experiment. They motivate the change but are not independently reproduced results for this branch or separate measurements of each optimization.
  • Cleanup is best effort. A timed-out target can retain Accessibility mode until a later successful capture or debugger detachment. The pending-read guard prevents queuing behind a known stuck read, and the new deadline bounds a disable request that itself stops responding. A deadline does not cancel Chrome's underlying command.

Thanks to @code-yeongyu for the report, measurements, and original prototype.

Maintainer review follow-up — 2026-10-23

Fixed the cleanup-ordering regression in 487296910bc7b88d2872add8cf2d9d808a670fb2; merged current main 8d00a2b03af06dc5668dda4cfcd8069c575f2d4f. Cleanup now follows all required reads. Its own timeout cannot invalidate captured root/child documents. Cancellation during cleanup still rejects promptly; already-pending reads and replacement attachments remain protected.

  • Extension suite: 2,922 passed, 254 skipped; TypeScript, production build, changed-file Biome and diff checks passed. Capture/read-gate suites: 77 passed.
  • Five out-of-tree regression cases exercise successful root/child cleanup timeouts, cancellation during either cleanup, and late replies after gate replacement. All pass. The child document is retained, and no capture reads follow cleanup.
  • Chrome for Testing 153.0.8010.12, isolated profile, actual captureObservationFacts and CdpReadGate: identical 1.5-second page work after AX succeeds on both base and fixed PR. Separately, a 1.5-second busy loop just before cleanup makes cleanup time out after about 1,000 ms while the capture still returns its document successfully.
  • Updated only the existing assertion/comment that required the incorrect ordering. No new tests, fixtures, scripts, screenshots, or logs were added to the repository. Reproduction code stays out of tree.

Reproduction procedure: capture a one-document page; inject a 1.5-second renderer busy loop after the final AX reply or after the final identity object-group release. Before the fix, the first case fails with Page.createIsolatedWorld, phase blocked; after the fix both retain the document. In the fixture, route every read and cleanup through the same production gate, hold only root/child Accessibility.disable, advance beyond one second, and assert all original documents remain. Abort during cleanup to check cancellation; replace its gate and settle the old promise to ensure it cannot release the new gate.

Hosted CI/Windows status is reported by the checks on the current commit; prior green checks are not substituted for these checks.

NianJiuZst and others added 5 commits October 8, 2026 12:19
The control overlay animated an inset box-shadow on a full-viewport element
forever. box-shadow is not compositor-animatable, so the whole viewport was
repainted on every frame for as long as the agent held control, even while it
was idle. The animation also ignored prefers-reduced-motion.

Paint the resting glow and the brighter glow as two static shadows and fade
only the brighter layer's opacity, for two pulses. Under reduced motion only
the resting glow is shown.

Refs Tencent#391.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LZRCYsFEKFsAZe6i8WAc9o
Every capture sent Accessibility.enable to each target but never disabled it,
so the renderer kept its accessibility tree live and reported changes for the
rest of the debugger attachment.

Once a target's AX trees are read, or the capture fails or is cancelled first,
disable the domain again on each target that enabled it. The cleanup shares
the capture's worker bound, settles before the capture does, only uses the
attachment that enabled AX, and is never queued behind a read still pending
past its deadline: such a target is skipped, and the read gate now refuses
Accessibility.disable behind a pending read.

Refs Tencent#391.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LZRCYsFEKFsAZe6i8WAc9o

@iuyo5678 iuyo5678 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the improvement. I found a reproducible capture regression in the Accessibility cleanup ordering that needs to be fixed before merging. Please address the inline comment and add regression tests to the repository for both root and child cleanup timeouts, while preserving the cancellation and late-reply safeguards.

},
signal,
);
).finally(releaseAx);

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P2] Run Accessibility cleanup after all required capture reads

This finally runs before document-identity verification and geometry collection. If Accessibility.disable exceeds its new 1-second deadline, releaseAccessibility catches the rejection, but CdpReadGate remains closed until the underlying command settles. The subsequent Page.createIsolatedWorld is therefore rejected as phase: "blocked". For the root target this fails the entire capture; for a child target, isolateChildTimeout discards the child's already-collected data. Catching the cleanup error alone does not preserve the capture result.

I reproduced the root failure using the actual captureObservationFacts and CdpReadGate code against an isolated Chrome 155.0.8059.85. Immediately after the final Accessibility.getFullAXTree response, I queued a 1.5-second Runtime.evaluate busy loop and allowed capture to continue without awaiting that loop:

  • Base e8777ab7: capture succeeded with one document in approximately 1,510 ms.
  • PR d305f8de: cleanup timed out after approximately 1,002 ms, and capture failed with CdpReadTimeoutError for Page.createIsolatedWorld, phase: "blocked".

A separate regression using the existing capture fixture and the real read gate also reproduces loss of the remote document when only the child's cleanup times out.

Please move the bounded cleanup after all required identity and geometry reads, for example into an outer finally covering the capture, so cleanup cannot invalidate an otherwise successful result. Keep cleanup on failure/cancellation, the original-attachment guard, and protection against pending reads.

Please add committed regression tests asserting that:

  1. A root cleanup timeout does not turn an otherwise successful capture into an error.
  2. A child cleanup timeout does not remove a fully captured child document.
  3. Cancellation still settles within the cleanup deadline, and a late cleanup reply releases only its original gate.

The existing tests cover a pending AX read blocking cleanup, but do not cover the effect of cleanup's own timeout on the remaining capture work.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Control overlay repaints the viewport every frame, and observe leaves the Accessibility domain enabled (GPU/CPU during sessions)

2 participants