feat(cli): cross-flavor inline image and video display via MCP and ACP#958
feat(cli): cross-flavor inline image and video display via MCP and ACP#958heavygee wants to merge 9 commits into
Conversation
Share display_image prompt across MCP-bridge flavors (Cursor, Gemini, Kimi, Codex, Claude, OpenCode), auto-approve the tool in buildHapiMcpBridge, handle ACP image content blocks, and harden generated-image registration with content sniffing. Closes tiann#956 Co-authored-by: Cursor <cursoragent@cursor.com>
Keep object URLs stable across refetch, upscale tiny inline images, fetch generated-image bytes with cache no-store (avoid empty 304 bodies), and load hapiMcpUrl from per-session API in hapi-display-image tooling. Co-authored-by: Cursor <cursoragent@cursor.com>
Add display_video alongside display_image, video MIME sniffing with avif guard, web GeneratedImageCard video player, and hapi-display-image auto-routing. Co-authored-by: Cursor <cursoragent@cursor.com>
Share display_video prompts across MCP-bridge flavors, auto-approve the tool, register mp4/webm via path sniffing, render inline video in web on the existing generated-image RPC path, and restore robust media card fetch. Co-authored-by: Cursor <cursoragent@cursor.com>
Bun hoists @modelcontextprotocol/sdk to the repo root; importing via cli/node_modules broke the dogfood script in worktrees. Co-authored-by: Cursor <cursoragent@cursor.com>
There was a problem hiding this comment.
Findings
- [Major] ACP inline media can be emitted before preceding assistant text. The new image branch flushes reasoning but leaves
bufferedTextopen, whileemitGeneratedImageFromAcpContent()emits asynchronously. For a normal ACP sequence like text chunk -> image block -> turn drain, the generated-image message can reach the web UI before the text that came first in the stream. Evidence:cli/src/agent/backends/acp/AcpMessageHandler.ts:559.
Questions
- None.
Summary
- Review mode: initial
- One ordering regression found in the latest diff. Residual risk: media fetch/cache behavior was reviewed statically only.
Testing
- Not run (automation). Suggested coverage: ACP handler test that sends a text
agentMessageChunk, then an imageagentMessageChunk, then drains, and asserts text precedesgenerated_image.
Flush buffered assistant text before async generated_image emit from ACP image blocks (PR tiann#958 review Major). Add optional source metadata on generated-image wire messages (ingress, flavor, toolCallId, toolName) for MCP, ACP, and Codex tool-result paths. Seeds artifact-event follow-up tiann#966. Co-authored-by: Cursor <cursoragent@cursor.com>
There was a problem hiding this comment.
Findings
- [Major] Generated media cards can keep rendering a stale blob after
imageIdchanges. - [Major] ACP image registration can let later updates render before the image block that preceded them.
Summary
Review mode: initial
Latest PR diff reviewed in full. The main residual risk is around generated media lifecycle/order because the new tests cover happy-path conversion and MIME helpers, but not component prop changes or ACP image blocks followed by later updates.
Testing
Not run (automation)
HAPI Bot
…tale blobs Queue ACP session updates and await async image registration before later events; clear GeneratedImageCard blob state when imageId changes. Co-authored-by: Cursor <cursoragent@cursor.com>
There was a problem hiding this comment.
Findings
- [Major] Await ACP updates that arrive during the late-drain window before emitting
turn_complete.cli/src/agent/backends/acp/AcpSdkBackend.ts:429
Summary
Review mode: follow-up after new commits
The previous stale-blob and initial ACP ordering findings are addressed. One ordering gap remains in the follow-up fix: late ACP image/file-backed updates can still be queued during drainLateBuffers() and emitted after turn_complete.
Testing
Not run (automation). Suggested: ACP prompt lifecycle test where a local-path image update arrives during drainLateBuffers() and must appear before turn_complete.
HAPI Bot
Straggler session/update during drainLateBuffers can queue async image registration; re-await sessionUpdateQueue so generated_image is not emitted after turn_complete. Co-authored-by: Cursor <cursoragent@cursor.com>
There was a problem hiding this comment.
Findings
- [Minor] AVIF files are now routed to
display_video, so the helper rejects a supported image format.scripts/tooling/hapi-display-image.mjs:32
Suggested fix:if (head.length >= 12 && head.subarray(4, 8).toString('ascii') === 'ftyp') { const brand = head.subarray(8, 12).toString('ascii') return brand === 'avif' || brand === 'avis' ? 'display_image' : 'display_video' }
Summary
Review mode: follow-up after new commits
One minor issue found in the helper script. The previous ACP ordering finding is addressed by the added post-drainLateBuffers() queue drain.
Testing
Not run (automation). Suggested: add a script/media sniffing test or shared helper coverage for AVIF vs MP4 ftyp routing.
HAPI Bot
Match server-side detectImageMimeType so .avif files are not sent to display_video and rejected as unsupported video. Co-authored-by: Cursor <cursoragent@cursor.com>
There was a problem hiding this comment.
Findings
No issues found.
Summary
Review mode: follow-up after new commits
The follow-up commit addresses the previous AVIF ftyp routing finding. I reviewed the full current diff across MCP/ACP media registration, generated-image RPC serving, web rendering, and helper tooling; no new high-confidence correctness, security, data-loss, or regression findings found. Residual risk: I did not run the full Bun test suite in this automation environment.
Testing
Not run (automation)
HAPI Bot
Summary
Cross-flavor inline image and video in HAPI web chat (not Cursor IDE composer):
display_imageanddisplay_videoinstartHappyServer; stdio bridge forwards both;buildHapiMcpBridgeauto-approves both tools.displayImagePrompt/display_videoexports for Claude, Codex, andHAPI_MCP_BRIDGE_PROMPT(Cursor, Gemini, Kimi, OpenCode first-prompt injection).generated-imageagent messages; generated media stored in CLI memory and served via hubGET /api/sessions/:id/generated-images/:imageId(images and video share this route).GeneratedImageCardrenders<img>(with tiny-image upscale + stable blob fetch) or<video controls>whenmimeTypeisvideo/*.hapi-display-image.mjsroutes mp4/webm todisplay_video(absolute paths); loadshapiMcpUrlfrom per-session GET.Test plan
bun typecheckgeneratedImages.test.ts,buildHapiMcpBridge.test.ts,codexMcpConfig.test.ts,AcpMessageHandler.test.ts,messageConverter.test.tsgeneratedInlineMedia.test.tsdisplay_imagePNG anddisplay_videoMP4 on Cursor session withhapiMcpUrl; inline cards in HAPI webIssues
Closes #956