Skip to content

feat(cli): cross-flavor inline image and video display via MCP and ACP#958

Open
heavygee wants to merge 9 commits into
tiann:mainfrom
heavygee:feat/cross-flavor-inline-images
Open

feat(cli): cross-flavor inline image and video display via MCP and ACP#958
heavygee wants to merge 9 commits into
tiann:mainfrom
heavygee:feat/cross-flavor-inline-images

Conversation

@heavygee

@heavygee heavygee commented Jun 19, 2026

Copy link
Copy Markdown
Collaborator

Summary

Cross-flavor inline image and video in HAPI web chat (not Cursor IDE composer):

  • CLI MCP: display_image and display_video in startHappyServer; stdio bridge forwards both; buildHapiMcpBridge auto-approves both tools.
  • Prompts: Shared displayImagePrompt / display_video exports for Claude, Codex, and HAPI_MCP_BRIDGE_PROMPT (Cursor, Gemini, Kimi, OpenCode first-prompt injection).
  • ACP: Image content blocks → generated-image agent messages; generated media stored in CLI memory and served via hub GET /api/sessions/:id/generated-images/:imageId (images and video share this route).
  • Web: GeneratedImageCard renders <img> (with tiny-image upscale + stable blob fetch) or <video controls> when mimeType is video/*.
  • Tooling: hapi-display-image.mjs routes mp4/webm to display_video (absolute paths); loads hapiMcpUrl from per-session GET.

Test plan

  • bun typecheck
  • CLI: generatedImages.test.ts, buildHapiMcpBridge.test.ts, codexMcpConfig.test.ts, AcpMessageHandler.test.ts, messageConverter.test.ts
  • Web: generatedInlineMedia.test.ts
  • Manual: display_image PNG and display_video MP4 on Cursor session with hapiMcpUrl; inline cards in HAPI web

Issues

Closes #956

heavygee and others added 2 commits June 19, 2026 17:54
Share display_image prompt across MCP-bridge flavors (Cursor, Gemini,
Kimi, Codex, Claude, OpenCode), auto-approve the tool in
buildHapiMcpBridge, handle ACP image content blocks, and harden
generated-image registration with content sniffing.

Closes tiann#956

Co-authored-by: Cursor <cursoragent@cursor.com>
Keep object URLs stable across refetch, upscale tiny inline images,
fetch generated-image bytes with cache no-store (avoid empty 304 bodies),
and load hapiMcpUrl from per-session API in hapi-display-image tooling.

Co-authored-by: Cursor <cursoragent@cursor.com>
heavygee and others added 3 commits June 20, 2026 22:01
Add display_video alongside display_image, video MIME sniffing with avif
guard, web GeneratedImageCard video player, and hapi-display-image auto-routing.

Co-authored-by: Cursor <cursoragent@cursor.com>
Share display_video prompts across MCP-bridge flavors, auto-approve the
tool, register mp4/webm via path sniffing, render inline video in web on
the existing generated-image RPC path, and restore robust media card fetch.

Co-authored-by: Cursor <cursoragent@cursor.com>
Bun hoists @modelcontextprotocol/sdk to the repo root; importing via
cli/node_modules broke the dogfood script in worktrees.

Co-authored-by: Cursor <cursoragent@cursor.com>
@heavygee heavygee changed the title feat(cli): cross-flavor inline image display via MCP and ACP feat(cli): cross-flavor inline image and video display via MCP and ACP Jun 20, 2026

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Findings

  • [Major] ACP inline media can be emitted before preceding assistant text. The new image branch flushes reasoning but leaves bufferedText open, while emitGeneratedImageFromAcpContent() emits asynchronously. For a normal ACP sequence like text chunk -> image block -> turn drain, the generated-image message can reach the web UI before the text that came first in the stream. Evidence: cli/src/agent/backends/acp/AcpMessageHandler.ts:559.

Questions

  • None.

Summary

  • Review mode: initial
  • One ordering regression found in the latest diff. Residual risk: media fetch/cache behavior was reviewed statically only.

Testing

  • Not run (automation). Suggested coverage: ACP handler test that sends a text agentMessageChunk, then an image agentMessageChunk, then drains, and asserts text precedes generated_image.

Comment thread cli/src/agent/backends/acp/AcpMessageHandler.ts Outdated
Flush buffered assistant text before async generated_image emit from ACP
image blocks (PR tiann#958 review Major). Add optional source metadata on
generated-image wire messages (ingress, flavor, toolCallId, toolName) for
MCP, ACP, and Codex tool-result paths. Seeds artifact-event follow-up tiann#966.

Co-authored-by: Cursor <cursoragent@cursor.com>

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Findings

  • [Major] Generated media cards can keep rendering a stale blob after imageId changes.
  • [Major] ACP image registration can let later updates render before the image block that preceded them.

Summary
Review mode: initial
Latest PR diff reviewed in full. The main residual risk is around generated media lifecycle/order because the new tests cover happy-path conversion and MIME helpers, but not component prop changes or ACP image blocks followed by later updates.

Testing
Not run (automation)

HAPI Bot

Comment thread web/src/components/AssistantChat/messages/ToolMessage.tsx
Comment thread cli/src/agent/backends/acp/AcpMessageHandler.ts Outdated
…tale blobs

Queue ACP session updates and await async image registration before later
events; clear GeneratedImageCard blob state when imageId changes.

Co-authored-by: Cursor <cursoragent@cursor.com>

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Findings

  • [Major] Await ACP updates that arrive during the late-drain window before emitting turn_complete. cli/src/agent/backends/acp/AcpSdkBackend.ts:429

Summary
Review mode: follow-up after new commits
The previous stale-blob and initial ACP ordering findings are addressed. One ordering gap remains in the follow-up fix: late ACP image/file-backed updates can still be queued during drainLateBuffers() and emitted after turn_complete.

Testing
Not run (automation). Suggested: ACP prompt lifecycle test where a local-path image update arrives during drainLateBuffers() and must appear before turn_complete.

HAPI Bot

Comment thread cli/src/agent/backends/acp/AcpSdkBackend.ts
Straggler session/update during drainLateBuffers can queue async image
registration; re-await sessionUpdateQueue so generated_image is not emitted
after turn_complete.

Co-authored-by: Cursor <cursoragent@cursor.com>

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Findings

  • [Minor] AVIF files are now routed to display_video, so the helper rejects a supported image format. scripts/tooling/hapi-display-image.mjs:32
    Suggested fix:
    if (head.length >= 12 && head.subarray(4, 8).toString('ascii') === 'ftyp') {
        const brand = head.subarray(8, 12).toString('ascii')
        return brand === 'avif' || brand === 'avis' ? 'display_image' : 'display_video'
    }

Summary
Review mode: follow-up after new commits
One minor issue found in the helper script. The previous ACP ordering finding is addressed by the added post-drainLateBuffers() queue drain.

Testing
Not run (automation). Suggested: add a script/media sniffing test or shared helper coverage for AVIF vs MP4 ftyp routing.

HAPI Bot

Comment thread scripts/tooling/hapi-display-image.mjs
Match server-side detectImageMimeType so .avif files are not sent to
display_video and rejected as unsupported video.

Co-authored-by: Cursor <cursoragent@cursor.com>

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Findings
No issues found.

Summary
Review mode: follow-up after new commits
The follow-up commit addresses the previous AVIF ftyp routing finding. I reviewed the full current diff across MCP/ACP media registration, generated-image RPC serving, web rendering, and helper tooling; no new high-confidence correctness, security, data-loss, or regression findings found. Residual risk: I did not run the full Bun test suite in this automation environment.

Testing
Not run (automation)

HAPI Bot

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

feat(cli+web): inline image display for all agent flavors

1 participant