Skip to content

Commit a4fec8a

Browse files
authored
feat: add MCP timeline MP4 rendering (#13)
* feat: add MCP timeline MP4 rendering Render authoritative Editor timelines through MCP, invalidate stale render pointers after edits, and keep existing Canvas timeline cards synchronized through the shared Command Kernel. * docs: document MCP timeline rendering Align the English, Chinese, and Japanese product documentation plus Agent integration skills with the 27-tool MCP catalog and the new authoritative MP4 export workflow. * docs: clarify MCP media runtime requirements Keep project guidance and third-party notices explicit that browser export stays FFmpeg-free while optional MCP rendering uses user-provided ffmpeg and ffprobe.
1 parent 2ab1998 commit a4fec8a

23 files changed

Lines changed: 898 additions & 34 deletions

File tree

‎AGENTS.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -40,7 +40,7 @@ Do not embed a proprietary general chatbot. BeatDesign must be complete without
4040
- Canvas layout snapshots are a UI persistence exception for drag, resize, and viewport state; external Agents still use semantic Canvas operations.
4141
- Agent changes must become visible in the browser workspace and remain inspectable, reversible where supported, and verifiable. A database revision alone is not proof of a successful user-visible operation.
4242
- Studio, Canvas, Editor, Assets, and MCP share project, task, asset, and generation services rather than duplicating business logic.
43-
- Preview and MP4 export use browser-native WebCodecs and Mediabunny. Do not make system FFmpeg a requirement for the core localhost UI. Node-side MCP frame extraction may use `ffmpeg` from `PATH` or `BEATDESIGN_FFMPEG` and must fail with a clear setup error when unavailable.
43+
- Preview and browser-side MP4 export use browser-native WebCodecs and Mediabunny. Do not make system FFmpeg a requirement for the core localhost UI. Node-side MCP frame extraction and Timeline rendering may use `ffmpeg` from `PATH` or `BEATDESIGN_FFMPEG`; Timeline rendering also uses `ffprobe` from `PATH` or `BEATDESIGN_FFPROBE`. These tools must fail with a clear setup error when a required binary is unavailable.
4444

4545
## Provider and storage boundary
4646

‎CHANGELOG.md‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -8,6 +8,7 @@ All notable changes to BeatDesign are documented in this file.
88

99
- Image overlays in the local Editor, with UI and MCP controls for placement, size, opacity, rotation, fades, and replacement with any project-owned image Asset.
1010
- Four caption style presets plus per-cue text, timing, size, width, and vertical-position controls shared by the UI and MCP command path.
11+
- MCP `bdesign_editor_render` support for rendering the authoritative Timeline to a project-owned MP4 with visible clips, overlays, caption burn-in, and mixed audio.
1112
- Japanese localization across the application and public READMEs.
1213

1314
### Changed
@@ -21,6 +22,7 @@ All notable changes to BeatDesign are documented in this file.
2122

2223
- Prepared connected local and generated references through the shared upload bridge before remote generation so providers receive public HTTPS media URLs.
2324
- Made static video previews seek past common opening black frames.
25+
- Invalidated stale Timeline renders after render-affecting UI or MCP edits and synchronized the current render state with an existing Canvas Timeline card.
2426

2527
## [0.2.2] - 2026-09-03
2628

‎README.ja.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -82,7 +82,7 @@ pnpm dev
8282

8383
## AgentからBeatDesignを使う
8484

85-
BeatDesignはProject、Asset、Canvas、生成、Editor操作をカバーする26個のローカルMCPツールを提供します。Agentによる変更は同じProjectサービスを通り、ブラウザーのワークスペースに表示されます。
85+
BeatDesignはProject、Asset、Canvas、生成、Editor操作をカバーする27個のローカルMCPツールを提供し、正式なTimelineからのMP4書き出しにも対応します。Agentによる変更は同じProjectサービスを通り、ブラウザーのワークスペースに表示されます。
8686

8787
MCP Hostへ接続した後は、次のように依頼できます。
8888

‎README.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -82,7 +82,7 @@ Add your own [BeatAPI API key](https://beatapi.io/dashboard/apikeys) only when y
8282

8383
## Use BeatDesign with an Agent
8484

85-
BeatDesign exposes 26 local MCP tools for Projects, Assets, Canvas, generation, and Editor operations. Agent changes use the same project services and become visible in the browser workspace.
85+
BeatDesign exposes 27 local MCP tools for Projects, Assets, Canvas, generation, and Editor operations, including authoritative MP4 timeline rendering. Agent changes use the same project services and become visible in the browser workspace.
8686

8787
After connecting your MCP host, you can ask:
8888

‎README.zh-CN.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -82,7 +82,7 @@ pnpm dev
8282

8383
## 让 Agent 操作 BeatDesign
8484

85-
BeatDesign 提供 26 个本地 MCP 工具,覆盖 Project、Asset、Canvas、生成和 Editor 操作。Agent 修改会经过同一套项目服务,并显示在浏览器工作空间中。
85+
BeatDesign 提供 27 个本地 MCP 工具,覆盖 Project、Asset、Canvas、生成和 Editor 操作,包括从权威时间线导出 MP4。Agent 修改会经过同一套项目服务,并显示在浏览器工作空间中。
8686

8787
连接 MCP Host 后,可以直接提出这样的要求。
8888

‎docs/MCP.md‎

Lines changed: 18 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -93,7 +93,7 @@ installation shapes.
9393

9494
## Tool groups
9595

96-
There are **26** tools:
96+
There are **27** tools:
9797

9898
- Project (5): list, get, create, target the current MCP session, and open a
9999
workspace review surface.
@@ -103,8 +103,8 @@ There are **26** tools:
103103
continue-from-tail-frame.
104104
- Generation (5): list model capabilities, read one model, submit an
105105
asset-first request, refresh status, and list history.
106-
- Editor (7): get, incremental edit, SRT import, semantic snapshot, diagnostics,
107-
deep-link view, and command history.
106+
- Editor (8): get, incremental edit, SRT import, authoritative MP4 render,
107+
semantic snapshot, diagnostics, deep-link view, and command history.
108108

109109
MCP writes use `origin=mcp` assigned inside the server. `canvas.apply` and
110110
`editor.apply` accept stable IDs, revisions, and idempotency keys. The server
@@ -157,6 +157,14 @@ Use `update_caption` to tune one caption cue's normalized font size, maximum
157157
width, and bottom position without changing later cues; `set_caption_style`
158158
continues to select the shared visual preset.
159159

160+
Use `bdesign_editor_render` to render the authoritative saved timeline to a
161+
project-owned MP4 Asset. The render includes visible video and image clips,
162+
image overlays, caption burn-in, and mixed audio. The tool requires `ffmpeg`
163+
and `ffprobe` on `PATH`, or explicit `BEATDESIGN_FFMPEG` and
164+
`BEATDESIGN_FFPROBE` paths. If the timeline changes while a render is running,
165+
the revision-checked commit rejects the stale output and removes that attempt's
166+
temporary Asset.
167+
160168
For a newly connected Canvas node, append a `place_card` operation after its
161169
`upsert_card`. By default it places the target once to the right of the frames
162170
listed in `sourceCardIds`, or to the right of the card's `referenceCardIds` when
@@ -171,12 +179,15 @@ generation independently of prompt text; BeatDesign does not insert synthetic
171179
- `bdesign_editor_snapshot` resolves active clips and source times; it does not
172180
rasterize a pixel frame yet.
173181
- `bdesign_asset_extract_frame` and `bdesign_canvas_continue_from_tail` decode
174-
the local video file. MCP/Node uses `ffmpeg` on PATH (or `BEATDESIGN_FFMPEG`);
175-
this is not a required system install for the browser UI.
182+
the local video file. MCP/Node frame extraction and timeline rendering use
183+
`ffmpeg` on PATH (or `BEATDESIGN_FFMPEG`); timeline rendering also uses
184+
`ffprobe` (or `BEATDESIGN_FFPROBE`). These are not required system installs
185+
for the browser UI.
176186
- SRT import validates the whole subtitle document before replacing the current
177187
caption track. Malformed input leaves the saved timeline unchanged.
178-
- Browser-only MP4 export is not exposed as a headless MCP tool yet. Caption
179-
burn-in is included when the browser exports an MP4.
188+
- Browser-native MP4 export remains available without system FFmpeg.
189+
`bdesign_editor_render` provides the corresponding MCP/Node export path and
190+
writes the result back as a project Asset.
180191
- `bdesign_asset_import` copies a local image, video, or audio file into the
181192
project Asset library from an absolute path. It does not place the Asset on
182193
Canvas or Editor; use `bdesign_canvas_apply` or `bdesign_editor_edit` after.

‎docs/PRODUCT_PLAN_AND_STATUS.md‎

Lines changed: 6 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -129,7 +129,7 @@ Codex / Claude Code / Other Agent
129129
### 媒体技术
130130

131131
- 使用 WebCodecs + Mediabunny 在浏览器完成媒体检查、编码和 MP4 封装。
132-
- 核心浏览器预览和 MP4 导出不要求系统 FFmpeg;可选的 MCP/Node 视频抽帧使用 `PATH` 中的 `ffmpeg` 或 `BEATDESIGN_FFMPEG`。
132+
- 核心浏览器预览和 MP4 导出不要求系统 FFmpeg;MCP/Node 视频抽帧与权威时间线 MP4 导出使用 `PATH` 中的 `ffmpeg`/`ffprobe`,也可通过 `BEATDESIGN_FFMPEG` 和 `BEATDESIGN_FFPROBE` 指定。
133133
- OpenReel 只作为时间线术语、文档模型和非破坏式编辑行为的参考;未打包其完整 UI 和应用外壳。
134134
- 媒体 metadata 采用限并发队列并带超时释放;单个损坏媒体不会永久阻塞后续卡片。
135135

@@ -144,7 +144,7 @@ Codex / Claude Code / Other Agent
144144
- Generation 的 `AssetFirstGenerationRequest` 已成为服务端权威输入:适配器媒体参数由 Asset ID 和当前 generation intent 编译,旧的客户端 URL 字段不再决定引用事实。
145145
- UI 命令入口不再接受客户端 `origin`;服务端固定写入 `ui`,MCP 入口在内核边界固定写入 `mcp`。
146146
- Provider Contract 已将逻辑模型目录与 BeatAPI effectId、上传路径和上游模型名拆开;BeatAPI 是默认实现,Fork 可在源码扩展点注册其他 Provider。
147-
- 本地 stdio MCP Server 提供 26 个工具(Project / Asset / Canvas / Generation / Editor),模型和参数通过 capability discovery 暴露;Canvas / Editor 增量操作使用完整 JSON Schema,Agent 可直接发现操作类型和参数。MCP 生成直接调用当前 Provider,本地产品不重复实现 API Key、余额、计费或限流策略,只透传 Provider 的结果与错误。`bdesign_project_target` 绑定当前会话项目,Project/Canvas/Editor view 工具返回 Codex Browser handoff;`bdesign_asset_import` 把本地文件导入项目 Asset 库;`bdesign_asset_extract_frame` 与 `bdesign_canvas_continue_from_tail` 负责抽帧续写;Editor MCP 可导入 SRT、放置和替换任意项目图片 Overlay,并调整叠层与单条字幕参数。
147+
- 本地 stdio MCP Server 提供 27 个工具(Project / Asset / Canvas / Generation / Editor),模型和参数通过 capability discovery 暴露;Canvas / Editor 增量操作使用完整 JSON Schema,Agent 可直接发现操作类型和参数。MCP 生成直接调用当前 Provider,本地产品不重复实现 API Key、余额、计费或限流策略,只透传 Provider 的结果与错误。`bdesign_project_target` 绑定当前会话项目,Project/Canvas/Editor view 工具返回 Codex Browser handoff;`bdesign_asset_import` 把本地文件导入项目 Asset 库;`bdesign_asset_extract_frame` 与 `bdesign_canvas_continue_from_tail` 负责抽帧续写;Editor MCP 可导入 SRT、放置和替换任意项目图片 Overlay、调整叠层与单条字幕参数,并通过 `bdesign_editor_render` 将权威时间线导出为项目内 MP4 Asset。
148148
- Codex、Claude Code 与 WorkBuddy 接入包内含 `beatdesign-workspace` Skill,负责项目选择、字幕/续写工具编排、付费生成停点和可视化复核;三者共用同一 MCP 与本地 Project 数据,其中 Claude Code 和 WorkBuddy 使用本机 HTTP MCP。
149149

150150
## 6. v0.2 Phase 1 本地已实现
@@ -159,7 +159,7 @@ Codex / Claude Code / Other Agent
159159
- Canvas -> Timeline Node -> Editor 连续工作流。
160160
- Editor 自动保存接入命令入口,并补齐冲突三方合并、重复操作保护和稳定播放头时间。
161161
- 图片 Clip、时间线拖拽调整持续时间与图片/视频统一视觉轨。
162-
- 本地 MCP Server 提供 26 个 Project、Asset、Canvas、Generation、Editor 工具;支持会话项目绑定、Canvas/Editor 可视化交接、从绝对路径导入本地素材、抽取尾帧续写、导入和精调 SRT 字幕,以及放置、替换和调整图片 Overlay。
162+
- 本地 MCP Server 提供 27 个 Project、Asset、Canvas、Generation、Editor 工具;支持会话项目绑定、Canvas/Editor 可视化交接、从绝对路径导入本地素材、抽取尾帧续写、导入和精调 SRT 字幕、放置/替换/调整图片 Overlay,以及权威时间线 MP4 导出。
163163
- MCP 增量 Canvas/Editor 命令在短暂 revision 竞争时会基于最新权威文档限次自动重放;持续冲突返回最新 revision 和明确重试提示。
164164
- Canvas 与 Editor 每 2 秒并在页面重新聚焦时检查 MCP 写入的新 revision。
165165

@@ -189,9 +189,9 @@ Codex / Claude Code / Other Agent
189189
- MCP Resources 和更完整的 schema versioning。
190190
- Agent Activity、命令审计和实时 UI 事件桥。
191191
- 外部市场正式审核与上架。仓库已提供 Codex 本地插件、可直接添加的 Claude Code 仓库插件市场,以及符合目录结构的 WorkBuddy MCP + Skill Connector;这些本地接入包不等于已通过第三方市场审核。
192-
- headless 预览/导出本地媒体 Worker。
192+
- 独立的 headless 像素预览与后台媒体 Worker;当前 MCP MP4 导出在本地 MCP Server 进程中完成。
193193

194-
当前可以称为“已支持本地 MCP 基础版”,但不能称为完整 Agent 编辑环境:像素级 Snapshot、headless 导出、实时 UI 事件和永久审计仍未实现。本地文件导入桥已完成;UI 当前使用 2 秒 revision 轮询和聚焦检查,而不是实时事件推送。
194+
当前可以称为“已支持本地 MCP 基础版”,但不能称为完整 Agent 编辑环境:像素级 Snapshot、独立后台媒体 Worker、实时 UI 事件和永久审计仍未实现。本地文件导入桥和 MCP 权威时间线 MP4 导出已完成;UI 当前使用 2 秒 revision 轮询和聚焦检查,而不是实时事件推送。
195195

196196
Canvas 的拖拽、缩放、视口和完整布局仍使用 revision-checked Snapshot 自动保存;Canvas/Timeline 业务 operation 已有 Command 合同,Timeline Node 回写也已接入 `/commands`。后续做 MCP parity 时,应继续把可语义化的 Canvas UI 动作迁移为 `canvas.apply`,不能让外部 Agent 调用完整 Snapshot 覆盖。
197197

@@ -200,7 +200,7 @@ Canvas 的拖拽、缩放、视口和完整布局仍使用 revision-checked Snap
200200
- Electron/Tauri 壳。
201201
- 内置 Node runtime、MCP 和媒体 Worker。
202202
- 原生文件选择、系统集成、签名、公证和自动更新。
203-
- 桌面封装阶段再评估是否内置 FFmpeg;当前开源 localhost 的核心 UI 不依赖它,但 MCP/Node 抽帧工具需要用户提供本地 `ffmpeg`。
203+
- 桌面封装阶段再评估是否内置 FFmpeg;当前开源 localhost 的核心 UI 不依赖它,但 MCP/Node 抽帧需要用户提供本地 `ffmpeg`,权威时间线导出还需要 `ffprobe`。
204204

205205
## 8. 路线图
206206

‎integrations/claude-code/beatdesign/skills/beatdesign-workspace/SKILL.md‎

Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -64,6 +64,19 @@ opened when the host cannot open it.
6464
`bdesign_canvas_apply`, then focus the same card with
6565
`bdesign_canvas_view`.
6666

67+
## Export the authoritative timeline
68+
69+
- When the user authorizes an export, call `bdesign_editor_get`, then call
70+
`bdesign_editor_render` with the returned revision. The result is a
71+
project-owned MP4 Asset containing visible clips, image overlays, caption
72+
burn-in, and mixed audio.
73+
- If rendering reports that `ffmpeg` or `ffprobe` is unavailable, stop and
74+
explain that the MCP process needs those binaries on `PATH`, or absolute
75+
`BEATDESIGN_FFMPEG` and `BEATDESIGN_FFPROBE` paths.
76+
- Render-affecting edits invalidate the previous current render without
77+
deleting its historical Asset. If the timeline changes while rendering,
78+
read the latest revision and ask before starting another render.
79+
6780
## Verify the visible result
6881

6982
After a write, read the returned revision and changed IDs. Canvas and Editor

‎integrations/codex/beatdesign/README.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -8,7 +8,7 @@ Canvas or Editor URL to Codex's in-app Browser for visible review.
88

99
```text
1010
Browser: pnpm dev → http://127.0.0.1:3020 (Canvas/Editor review surface)
11-
Agent: MCP stdio → pnpm mcp (Agent calls 26 tools)
11+
Agent: MCP stdio → pnpm mcp (Agent calls 27 tools)
1212
Both processes share the same local SQLite + project files.
1313
```
1414

‎integrations/codex/beatdesign/skills/beatdesign-workspace/SKILL.md‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -39,6 +39,12 @@ For Canvas-specific review, call `bdesign_canvas_view` with `cardId` so the work
3939
- After submission, follow generation status until it succeeds, fails, or needs user action. A successful generation creates an Asset but does not place itself. Read the current continuation card, preserve its generation settings, update the returned `generationCardId` with that output through `bdesign_canvas_apply`, then call `bdesign_canvas_view` with the same card ID.
4040
- If continuation returns `ok=false`, preserve its structured conflict or rollback result. Follow a returned retry instruction at most once after reading current state; do not blindly re-extract frames.
4141

42+
## Export the authoritative timeline
43+
44+
- When the user authorizes an export, call `bdesign_editor_get`, then call `bdesign_editor_render` with the returned revision. The result is a project-owned MP4 Asset that includes visible clips, image overlays, caption burn-in, and mixed audio.
45+
- If rendering reports that `ffmpeg` or `ffprobe` is unavailable, stop and explain that the MCP host needs those binaries on `PATH`, or absolute `BEATDESIGN_FFMPEG` and `BEATDESIGN_FFPROBE` paths.
46+
- A render-affecting edit invalidates the previous current render without deleting its historical Asset. If the timeline changes during rendering, read the latest revision and ask before starting another potentially expensive render.
47+
4248
## Verify what the user can see
4349

4450
After a write, read the returned revision and changed IDs, then inspect the already-open Canvas or Editor. Canvas and Editor currently refresh external revisions within about two seconds and on focus. Confirm the intended card, clip, caption, duration, or media is visible; a successful database revision alone is not completion.

0 commit comments

Comments
 (0)