Repository navigation
berd-call: download and preview Siri voices, finish speech across restarts, and steady the Spokesperson - #416
johnmatthewtennant wants to merge 37 commits into
Conversation
Signed-off-by: Nora <49aa1f65411fd096d2e2ec144f1e7aa36fdc76d1b907cfdf7be000c66f9d3b8e@buzz.block.builderlab.xyz>
Signed-off-by: Nora <49aa1f65411fd096d2e2ec144f1e7aa36fdc76d1b907cfdf7be000c66f9d3b8e@buzz.block.builderlab.xyz>
Siri renders faster than playback drains, and the native callback failed the whole reply once 60 seconds of audio were pending. It now waits for room when the bound is full and keeps it as a memory cap. Signed-off-by: John Tennant <jtennant@block.xyz> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Nora <49aa1f65411fd096d2e2ec144f1e7aa36fdc76d1b907cfdf7be000c66f9d3b8e@buzz.block.builderlab.xyz>
The menu bar's voice submenu listed only installed Siri voices, so a voice had to be downloaded with `berd-call voices download` before it could be chosen. The submenu now offers the uninstalled voices in the call's language under Download, with their sizes. Choosing one downloads it off the main thread, shows its progress, and switches the call to it once it is installed; a failure shows as a control error. Signed-off-by: John Tennant <jtennant@block.xyz> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A voice whose catalog name is a code, such as en-US-F, now takes its display name from the build paths in its installed asset's Info.plist (.../assets/zack_v2_dual_prompt_with_fallback/p2a is Zack). The known names and "Siri Voice N" remain the fallbacks. Signed-off-by: John Tennant <jtennant@block.xyz> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Berd sets the Spokesperson's realtime session options (model, turn detection, noise reduction and the rest of its Advanced settings), but the standalone CLI always used the defaults. `--spokesperson-options JSON` now takes the same camelCase fields and ranges, refuses unknown ones and the voice and speed that have their own flags, and applies them when the Spokesperson's session starts. Saved settings don't keep it; the caller passes it on each start. Signed-off-by: John Tennant <jtennant@block.xyz> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Rolling back a Spokesperson voice change blocked the call until OpenAI confirmed the cutover's input clear, and stalled the menu bar when that confirmation was slow. Abandon the cutover immediately instead and attribute its late confirmation by send order, so it can neither complete a later reset nor report a stale cutover result. Retire the candidate runtime in the background. Signed-off-by: John Tennant <jtennant@block.xyz> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Now that aborting a cutover no longer waits for the worker, the frames a rolled-back voice change replays could reach the provider before the cutover's clear and be discarded with it. Send cutover commands on the input queue, like resets, so they stay in order with the audio they clear. Signed-off-by: John Tennant <jtennant@block.xyz> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Nora <49aa1f65411fd096d2e2ec144f1e7aa36fdc76d1b907cfdf7be000c66f9d3b8e@buzz.block.builderlab.xyz>
Signed-off-by: Nora <49aa1f65411fd096d2e2ec144f1e7aa36fdc76d1b907cfdf7be000c66f9d3b8e@buzz.block.builderlab.xyz>
When the Realtime session was lost or expired while an input reset was waiting on the provider's confirmation, the call ended with "Spokesperson runtime is closed" instead of starting a replacement session. The lost session takes its input buffer with it, so treat its outstanding clears as done once its terminal event arrives, answer the waiting input controls, and let recovery proceed. Signed-off-by: John Tennant <jtennant@block.xyz> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A mute or input reset waits to be acknowledged until the provider has cleared its input. When the host shut the session down, or closed its input, while a clear was outstanding, the session exited without ever answering those controls. Wait for the outstanding clears, which the runtime settles within its acknowledgement timeout, and answer them before the session ends. Signed-off-by: John Tennant <jtennant@block.xyz> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* https://github.com/block/berd: Verify Auto muting follows real macOS output changes berd-call: persist Auto input muting and follow output changes
A restart replaced the saved arguments with the new session's, so turning the Spokesperson off (a mode that can't carry --spokesperson-voice and --spokesperson-rate) erased them, and turning it back on started with the default voice. Keep them as call start does, and start a restart that can carry them with the saved voice unless it sets its own. Signed-off-by: John Tennant <jtennant@block.xyz> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Realtime, speech to text and text to speech each read their own key (OPENAI_REALTIME_API_KEY, OPENAI_STT_API_KEY, OPENAI_TTS_API_KEY) and fall back to OPENAI_API_KEY when it is unset or blank. Error messages, public startup errors and the STT benchmark credential_source name the variable that applies. Signed-off-by: John Tennant <jtennant@block.xyz> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: John Tennant <jtennant@block.xyz> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: John Tennant <jtennant@block.xyz> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Auto resolved with no device name for every CLI call, and the shared classifier only applied its speaker-name check to a named device. A default output called "USB Speakers" or a Bluetooth speaker was judged not-speakers, so Auto allowed microphone input during playback and the call could hear itself. With no name, look up the default output device's name (CoreAudio) and apply the same name check, keeping the built-in speaker metadata as the fallback. The composition takes injected name and metadata lookups so the classification is tested without audio hardware. Signed-off-by: John Tennant <jtennant@block.xyz> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
macOS 27 removed UAFAssetUtilitiesService from UnifiedAssetFramework, so every voice download failed with "Siri voice download service is unavailable." The service now lives in SiriAssetUtilities as SAUAssetUtilitiesService, with the same switchLanguage: and downloadSiriAssets methods. Look for it there first and keep the old class for earlier releases. On macOS 27.0.1, `berd-call voices download --voice Simone --language en-US` failed with the old lookup; with this one, `--voice Quinn` downloads and reports installed in 19 seconds. Signed-off-by: John Tennant <jtennant@block.xyz> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Hosts such as the Buzz plugin need to let a user hear a Siri voice before choosing it, but berd-call could only list and download voices. Previewing lived only in Berd's app command. `berd-call voices preview --voice NAME --language BCP47 [--rate R]` follows the app's preview: Apple's system sample plays first, since it works for voices that are not downloaded, and an installed voice speaks a short fixed sentence only when no sample plays. Playback uses the default output device and the command exits when it ends, printing one `voices.preview` result envelope with the voice, rate, and whether a `sample` or `speech` played. Rate defaults to 1.0 and is bounded to 0.5..2.0 like the other Siri commands. SIGINT and SIGTERM stop playback through the bridge's stop callback and exit 130 without a result, so a host can end a preview by killing the process. The playback logic is a reusable `siri::preview_voice` beside `download_voice`. Berd's app command is left unchanged; it can adopt the function later, but that change could not be validated in this berd-call-only build. Signed-off-by: John Tennant <jtennant@block.xyz> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Downloading a Siri voice evicted the voice downloaded before it, and the voice it downloaded then listed as not installed. Eviction: `voices download` subscribed every voice under the one client id com.apple.speech. sirittsd keeps a single subscription per client id and a new one replaces the last, unsubscribing its voice from the com.apple.siri.tts asset set (its log shows subscribe Damon and unsubscribe Simone, then subscribe Quinn and unsubscribe Simone). Apple's own clients keep their voices because each holds its own client id. Each voice is now subscribed under its own id, so a download only adds. Stale list: `download` trusted sirittsd's downloadedVoicesMatching:, while `list` also requires an existing TTSAXResourceManager contentPath. axassetsd's voice database stays stale after a download made outside System Settings, so a voice that was downloaded, on disk and speaking (Damon) listed as not installed, and download returned success for it without changing that. The content check now falls back to the voice's UAF asset on disk, matched by the CFBundleName in its Info.plist, which is current right after a download. Download and its polling use the same test as list (sirittsd downloaded and assets on disk), so en-GB-C, which sirittsd claims but has no assets, is still not installed and is now fetched instead of reported done. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: John Tennant <jtennant@block.xyz>
Turning the Spokesperson on or off restarts the voice session, and the restart sent Shutdown at once, which cancelled the reply being played and failed the speaks queued behind it with "voice session restarted". A restart now waits until no speech is playing or queued, then shuts the session down, like voice and speed changes that take effect between replies. The wait is bounded at 60 seconds. A second restart while one waits is refused, and stopping the call still cuts speech off at once and cancels a waiting restart. Signed-off-by: John Tennant <jtennant@block.xyz> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A normal Spokesperson reply plays without any Speak command, so pending_speak and waiting_speaks are empty and a restart (such as turning the Spokesperson off) cut it off. The audio host now reports whether it is playing speech, and a restart waits for that too. The 60 s fallback still applies, and the flag clears when the audio host thread ends, so a dead session cannot hold a restart. Signed-off-by: John Tennant <jtennant@block.xyz> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
When a voice has no system sample, `voices preview` spoke a sentence after `validate_installed_voice`, which asks sirittsd alone. sirittsd can claim a voice whose assets are missing and then speak another voice, so the preview could play the wrong voice and report `source: speech` for the requested one. Speak only when the catalog lists the exact voice as installed, the same test `voices list` and `voices download` use. Signed-off-by: John Tennant <jtennant@block.xyz> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The menu bar's "Spokesperson" checkmark didn't say what it turns on. It is now "Enable Experimental Realtime Spokesperson", and it sits at the bottom of the menu with the Spokesperson's voice, just above End Call. Signed-off-by: John Tennant <jtennant@block.xyz> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
johnmatthewtennant
changed the title
johnmatthewtennant
changed the title
Signed-off-by: John Tennant <jtennant@block.xyz> Co-authored-by: Aaron <f425b86c3b780d43f8d9cf1802429d04dd985bf92eac44fca9468b5ad3887c5e@buzz.block.builderlab.xyz> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
johnmatthewtennant
changed the title
Co-authored-by: Aaron <f425b86c3b780d43f8d9cf1802429d04dd985bf92eac44fca9468b5ad3887c5e@buzz.block.builderlab.xyz> Signed-off-by: John Tennant <jtennant@block.xyz>
Request write readiness so Darwin reports a pipe reader closing, and throttle writable polls to avoid spinning. Co-authored-by: Nora <49aa1f65411fd096d2e2ec144f1e7aa36fdc76d1b907cfdf7be000c66f9d3b8e@buzz.block.builderlab.xyz> Signed-off-by: John Tennant <jtennant@block.xyz>
morgmart
left a comment
morgmart
left a comment
There was a problem hiding this comment.
🤖 Automated code review
REQUEST_CHANGES. Fresh static Engineering review of the exact three-dot comparison found three blocking defects: desktop Automatic interruption mode is resolved to a fixed policy before the live route monitor can follow output changes; the new Spokesperson loudness processing is wired only into the managed desktop Realtime host rather than the berd-call Expert-Spokesperson playback path; and restart can still observe audio_playing as false while Spokesperson audio is entering the audio host, then shut down and cut off that reply. Supplied GitHub evidence is structurally valid: nine checks are successful and the macOS Rust check is in progress at the reviewed head SHA; required checks independently govern merge readiness. Final self-check covered every changed call, restart, muting, download, preview, Spokesperson, credential, persistence, and lifecycle flow; accessibility; localization; navigation and consent guards; async, error, never-completes, lifecycle, and race behavior; test honesty; project rules; duplicate and overlapping findings; and concrete evidence plus user effect for blocking severity.
Deterministic publication result: 3 blocking and 0 non-blocking inline finding(s) publishable; 0 duplicate(s) suppressed; 0 blocking screenshot-evidence requirement(s) in this review body.
| VoiceInterruptionMode::Automatic | VoiceInterruptionMode::AllowInterruptions => { | ||
| InputDuringTtsPolicy::AllowBargeIn | ||
| } | ||
| VoiceInterruptionMode::Automatic => InputDuringTtsPolicy::Auto |
There was a problem hiding this comment.
🤖 P1 · Keep Automatic policy live (blocking)
Automatic is immediately resolved to AllowBargeIn or SuppressInput using the output route observed when playback starts. VoiceInputControls starts its new route-monitor thread only for an unresolved Auto activity, so the desktop Siri, Pocket, and OpenAI playback paths never activate that monitor and cannot follow an output-device change during speech.
User effect: If someone starts speech on headphones and switches to speakers while it is playing, microphone input can remain enabled and feed the assistant's audio back into the call instead of muting automatically.
Recommended fix: Return InputDuringTtsPolicy::Auto for VoiceInterruptionMode::Automatic and let VoiceInputControls resolve and monitor the current route. Keep the explicit interruption modes mapped to their fixed policies.
Test: Add a desktop playback regression that starts an Automatic Siri, Pocket, or OpenAI assistant activity on headphones, changes the reported route to speakers while the activity remains active, and verifies effective mute changes. Assert that the admission path receives Auto; the current implementation should fail before the route change assertion.
| samples.len() as u64, | ||
| false, | ||
| )?; | ||
| active.loudness.process(&mut samples); |
There was a problem hiding this comment.
🤖 P1 · Level berd-call playback too (blocking)
The new gain and limiter are applied only in RealtimePlaybackHost, which is used by the managed desktop OpenAI Realtime command. The berd-call Expert-Spokesperson session has a separate AudioDelta to LivePlayback pipeline in main.rs and still forwards the provider samples unchanged. Consequently the production path this PR describes and the Buzz plugin consumes does not receive the advertised loudness compensation.
User effect: Spokesperson replies in Berd Call remain noticeably quieter than Siri even though this change claims to normalize them, so people still experience inconsistent call volume.
Recommended fix: Apply the shared SpokespersonLoudness processor to each berd-call live response before samples enter its playback queue, preserving one processor across chunks for that response. Keep both playback paths on the same shared implementation.
Test: Add an integration-level berd-call live-playback test that sends quiet and near-ceiling AudioDelta chunks through the Expert-Spokesperson session path and asserts the audio pipe receives boosted, capped samples with state preserved across chunks. The current CLI path should emit the original samples.
| if let Some((speech_id, played_frames)) = finished { | ||
| playback.finish(speech_id, played_frames); | ||
| } | ||
| playing.0.store( |
There was a problem hiding this comment.
🤖 P1 · Synchronize restart audio ingress (blocking)
The session actor decides that restart is quiet from pending speak work plus an atomic audio_playing snapshot, while the audio-host thread publishes that snapshot only after reading and processing available audio records. A spontaneous Spokesperson reply has no pending_speak entry, so a restart can read false while an audio-begin record is already entering the host and send Shutdown before the host publishes true. The added tests set audio_playing directly and do not exercise this interleaving.
User effect: Changing a setting or toggling the Spokesperson at the start of a reply can cut off that reply even though restart is supposed to wait for playing and queued speech.
Recommended fix: Use an ordered drain handshake between the restart actor and audio host. Before sending Shutdown, establish that all audio records ordered before the restart have been consumed and playback is drained, while retaining the 60-second bound for a host that never drains.
Test: Add a deterministic concurrency regression that pauses the audio host after it accepts a Spokesperson audio-begin record but before audio_playing becomes visible, requests restart, and verifies Shutdown is withheld until that audio drains. The current implementation should send Shutdown during the pause.
This branch has not been deployed
2 participants
Summary
The Buzz Berd Call plugin uses
berd-callfor voice calls in Buzz. These calls need speech that finishes playing, input muting that follows the output device, Siri voices people can download and audition, and a Spokesperson whose settings survive restarts. The plugin's voice preview and its advanced Spokesperson options require this PR. Olderberd-callbuilds have neither.Speech playback
Input muting
Siri voices
berd-call voices preview --voice NAME --language BCP47 [--rate R]plays Apple's sample. When a voice has no sample, it speaks with that voice if its assets are on disk. Stopping the preview process stops playback.Realtime Spokesperson (experimental)
--spokesperson-optionssets advanced session options.OPENAI_REALTIME_API_KEY,OPENAI_STT_API_KEY,OPENAI_TTS_API_KEY), falling back toOPENAI_API_KEY.Related issue
Combines and supersedes #403 and #405 through #415, which retain their original validation evidence.
Testing
exceeded its bounded PCM queuebefore the fix and plays to completion after it.berd-call synthesizeagainst a local server sent the service key when set, the shared key when it was unset or blank, and the documented error when neither existed.Screenshots of the changed menu bar surfaces (Voice ▸ Download and the renamed Spokesperson toggle) are still to come.