Skip to content

fix(extension): reduce idle overlay and Accessibility overhead - #408

Open
NianJiuZst wants to merge 5 commits into
Tencent:mainfrom
NianJiuZst:fix/391-idle-overlay-ax-cost
Open

NianJiuZst wants to merge 5 commits into
Tencent:mainfrom
NianJiuZst:fix/391-idle-overlay-ax-cost

Conversation

@NianJiuZst

@NianJiuZst NianJiuZst commented Oct 8, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Fixes #391.

An agent session can keep doing browser work between commands: the control overlay continuously animates a viewport-sized inset shadow, and observation enables Chrome's Accessibility domain without disabling it afterward. This PR makes the control glow settle after two opacity pulses and releases Accessibility on the targets whose capture reads have finished.

Problem and resulting behavior

Area Before After
Control glow An infinite box-shadow animation runs for the entire time the overlay is visible. Two static shadow layers are painted; only the brighter layer's opacity animates, for two 3-second cycles, before the resting glow remains.
Reduced motion The breathing animation ignores the user's motion preference. prefers-reduced-motion: reduce disables the pulse and leaves the static control glow.
Observation Each target receives Accessibility.enable, but completed captures leave the domain enabled. Successfully enabled targets receive Accessibility.disable after the AX read phase, before document-identity and geometry work.
Capture interruption There is no capture-scoped Accessibility release. Failure or cancellation triggers release for the targets already enabled, while attachment and pending-read checks protect cleanup dispatch.

Implementation

Control overlay

ControlOverlay.tsx keeps the resting inset shadow on the existing overlay and adds a brighter child layer. The bsk-breathe keyframes now contain only opacity changes, with a finite iteration count of two. The child has a resting opacity of zero, and a reduced-motion media query disables its animation. The control indicator, interruption button, and input-blocking behavior remain available.

Accessibility lifetime

capture-coordinator.ts records each successfully enabled target together with its debugger attachment ID. Release uses the capture's existing worker helper, with at most four cleanup requests in flight, and is awaited before the capture completes.

  • Release runs when target collection fails or is cancelled, and in finally around the AX read phase.
  • Cleanup uses sendGuarded with the original attachment ID in production, so an old capture cannot reconnect or send cleanup to a replacement attachment.
  • A target whose AX read times out is removed from the release set because the underlying Chrome command can still be running.
  • command-deadline.ts guards Accessibility.disable against earlier pending reads and gives cleanup itself a 1-second deadline. A timed-out disable retains its target gate until Chrome replies or the debugger detaches; late replies cannot clear a replacement attachment.
  • Rejected cleanup requests are treated as best-effort cleanup and do not replace the capture result or its original error.

Observation refs retain their backend DOM node IDs and document/attachment identity; the implementation does not depend on Accessibility node IDs surviving between captures. The changelog records both user-visible changes.

Regression coverage

Eight new cases extend three existing test files:

  • ControlOverlay.test.tsx (2): static shadows, opacity-only keyframes, exactly two pulses, no infinite animation, and the reduced-motion override.
  • capture-coordinator.test.ts (5): release for root and child targets after their AX reads; release before identity/geometry work; dispatch on the original attachment; no cleanup on a replacement attachment; skipping a timed-out child while releasing the root; and cleanup during cancellation. Some cases assert multiple related behaviors.
  • command-deadline.test.ts (1): a pending child-session AX read blocks its own cleanup without blocking root-target cleanup.

Validation — 2026-10-20

Merged upstream main e8777ab71bb851196895515b53eea2481462dd57; current revision d305f8de8867bc42081d23c765502ab2291ce44b.

  • Extension suite: 2,683 passed, 244 skipped. The three directly affected test files pass 82 tests.
  • Extension TypeScript, changed-file Biome, production build and diff checks passed.
  • Two additional out-of-tree capture regressions cover completion and cancellation with a stuck child cleanup. Both settle within the cleanup deadline; the root remains usable, the affected child stays gated, and its late response restores access.
  • Real Chrome 153: after AX reads, an intentionally blocked renderer kept Accessibility.disable pending. The production read gate returned its timeout after 1,002 ms, rejected another read while pending, and recovered after the original command settled following execution termination.
  • Independent full-PR P0–P2 review passed. New proof scripts/logs remain outside the repository.

Limits and performance evidence

  • Capture regressions use simulated CDP; the blocked-renderer deadline was also checked against real Chrome. CPU/GPU profiling was not rerun; skipped tests are not counted as passes.
  • The measurements in issue #391 belong to the reporter's combined overlay/Accessibility experiment. They motivate the change but are not independently reproduced results for this branch or separate measurements of each optimization.
  • Cleanup is best effort. A timed-out target can retain Accessibility mode until a later successful capture or debugger detachment. The pending-read guard prevents queuing behind a known stuck read, and the new deadline bounds a disable request that itself stops responding. A deadline does not cancel Chrome's underlying command.

Thanks to @code-yeongyu for the report, measurements, and original prototype.

NianJiuZst and others added 5 commits October 8, 2026 12:19
The control overlay animated an inset box-shadow on a full-viewport element
forever. box-shadow is not compositor-animatable, so the whole viewport was
repainted on every frame for as long as the agent held control, even while it
was idle. The animation also ignored prefers-reduced-motion.

Paint the resting glow and the brighter glow as two static shadows and fade
only the brighter layer's opacity, for two pulses. Under reduced motion only
the resting glow is shown.

Refs Tencent#391.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LZRCYsFEKFsAZe6i8WAc9o
Every capture sent Accessibility.enable to each target but never disabled it,
so the renderer kept its accessibility tree live and reported changes for the
rest of the debugger attachment.

Once a target's AX trees are read, or the capture fails or is cancelled first,
disable the domain again on each target that enabled it. The cleanup shares
the capture's worker bound, settles before the capture does, only uses the
attachment that enabled AX, and is never queued behind a read still pending
past its deadline: such a target is skipped, and the read gate now refuses
Accessibility.disable behind a pending read.

Refs Tencent#391.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LZRCYsFEKFsAZe6i8WAc9o

@iuyo5678 iuyo5678 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the improvement. I found a reproducible capture regression in the Accessibility cleanup ordering that needs to be fixed before merging. Please address the inline comment and add regression tests to the repository for both root and child cleanup timeouts, while preserving the cancellation and late-reply safeguards.

},
signal,
);
).finally(releaseAx);

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P2] Run Accessibility cleanup after all required capture reads

This finally runs before document-identity verification and geometry collection. If Accessibility.disable exceeds its new 1-second deadline, releaseAccessibility catches the rejection, but CdpReadGate remains closed until the underlying command settles. The subsequent Page.createIsolatedWorld is therefore rejected as phase: "blocked". For the root target this fails the entire capture; for a child target, isolateChildTimeout discards the child's already-collected data. Catching the cleanup error alone does not preserve the capture result.

I reproduced the root failure using the actual captureObservationFacts and CdpReadGate code against an isolated Chrome 155.0.8059.85. Immediately after the final Accessibility.getFullAXTree response, I queued a 1.5-second Runtime.evaluate busy loop and allowed capture to continue without awaiting that loop:

  • Base e8777ab7: capture succeeded with one document in approximately 1,510 ms.
  • PR d305f8de: cleanup timed out after approximately 1,002 ms, and capture failed with CdpReadTimeoutError for Page.createIsolatedWorld, phase: "blocked".

A separate regression using the existing capture fixture and the real read gate also reproduces loss of the remote document when only the child's cleanup times out.

Please move the bounded cleanup after all required identity and geometry reads, for example into an outer finally covering the capture, so cleanup cannot invalidate an otherwise successful result. Keep cleanup on failure/cancellation, the original-attachment guard, and protection against pending reads.

Please add committed regression tests asserting that:

  1. A root cleanup timeout does not turn an otherwise successful capture into an error.
  2. A child cleanup timeout does not remove a fully captured child document.
  3. Cancellation still settles within the cleanup deadline, and a late cleanup reply releases only its original gate.

The existing tests cover a pending AX read blocking cleanup, but do not cover the effect of cleanup's own timeout on the remaining capture work.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Control overlay repaints the viewport every frame, and observe leaves the Accessibility domain enabled (GPU/CPU during sessions)

2 participants