Narrated walkthrough: in-page play mode + opt-in MP4 export#2
Open
russ wants to merge 2 commits into
Open
Conversation
Turn the static walkthrough into a self-playing, narrated screencast: each chapter becomes a scene that pans the actual diff and spotlights the lines it references, while the browser's built-in speech synthesis reads a short narration (captions included, so it works muted). No new dependencies, no API key, no network -- the same single .html, just playing itself. - core: add optional Chapter.narration and allow it in the JSON Schema. Falls back to intent then summary, so existing walkthroughs still play. - web: player overlay -- requestAnimationFrame scene clock, speech + auto-advance, transport controls, diff spotlight/dim, keyboard (space/arrows/m/Esc), header Play button and `p` shortcut. - styles: full-screen player themed via the existing light/dark CSS vars, with a prefers-reduced-motion fallback. - skill + README: author a per-chapter `narration`; document play mode. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01M8q2ot94CPocoNHL7XVStW
An opt-in counterpart to the in-page play mode that produces a real,
shareable .mp4 -- a title card plus one scene per chapter, each panning the
actual diff (spotlighting referenced lines) over a narration track. Reuses
the same scene model, but uses *system* tools, so it adds no npm runtime
dependencies and only touches them when a video is requested.
Pipeline per scene: build a deterministic scene HTML (fixed header/caption
bands) -> headless-Chromium screenshot -> TTS narration -> ffmpeg slices the
PNG into a fixed header, a vertically-panning code region, and a fixed
caption, then muxes the audio. Per-scene clips are concatenated to the MP4.
- renderer: packages/renderer/src/video/{scene-html,index}.ts; export
renderVideo + types.
- cli: `patchstory video <walkthrough.json>` with
--tts/--voice/--chrome/--fps/--keep (plus --diff/--redact via render path).
- tools: resolve ffmpeg/ffprobe by actually running candidates (PATH, then
/usr/bin, then PATCHSTORY_FFMPEG/FFPROBE), so a broken or shadowing PATH
entry is skipped; Chrome resolution also picks up a flatpak Chromium.
- tts: elevenlabs (ELEVENLABS_API_KEY) | espeak-ng | flite | say | none.
- docs: README "Narrated video" section; skill note.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01M8q2ot94CPocoNHL7XVStW
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What & why
Turns a PatchStory walkthrough into a narrated screencast that breaks down the code being reviewed — inspired by this AI-video-generation pipeline, but built to fit PatchStory's local-first, zero-dependency model. The walkthrough's chapters already are the script (intent, risk, the referenced diff hunks), so this mostly adds a player and an exporter over data that already exists.
Two layers:
1. In-page "play" mode (the default — stays zero-dependency)
A ▶ Play button turns the static
.htmlinto a self-playing screencast: each chapter becomes a scene that pans the actual diff and spotlights the lines it references, while the browser's built-in speech synthesis reads a short narration (captions included, so it works muted). No ffmpeg, no API key, no network — the same single.html, just playing itself.Chapter.narrationfield (falls back tointent→summary, so existing walkthroughs still play).requestAnimationFramescene clock, speech + auto-advance, transport controls, diff spotlight/dim, keyboard (space/←/→/m/Esc), header Play button andpshortcut.prefers-reduced-motionfallback.2.
patchstory video(opt-in MP4 export)For when you need a real shareable file. Same scene model, rendered to an
.mp4using system tools (no npm runtime deps; only touched when you ask for a video):Per scene: deterministic scene HTML (fixed header/caption bands) → headless-Chromium screenshot → TTS → ffmpeg slices the screenshot into a fixed header, a vertically-panning code region, and a fixed caption, then muxes audio. Clips are concatenated to the final MP4.
/usr/bin→PATCHSTORY_FFMPEG/FFPROBE), so a broken/shadowing PATH entry is skipped.--chrome,PATCHSTORY_CHROME, or a flatpak Chromium.--tts:elevenlabs(ELEVENLABS_API_KEY), or localespeak-ng/flite/say, ornone(silent; captions still shown).Verification
typecheckclean ·buildclean · 23/23 tests pass.onerrorwas instantly skipping scenes instead of falling back to the timed clock.patchstory videoproduces a valid 1920×1080 H.264 + AAC, ~133s MP4 from a 5-chapter walkthrough; sampled frames confirm the title card, panning syntax-highlighted diff with spotlit lines, and burned-in captions.Notes
🤖 Generated with Claude Code