diff --git a/src-tauri/crates/berd-call/README.md b/src-tauri/crates/berd-call/README.md index 50779fb21..f7bc2ce57 100644 --- a/src-tauri/crates/berd-call/README.md +++ b/src-tauri/crates/berd-call/README.md @@ -162,7 +162,7 @@ berd-call session --tts-backend openai --tts-url https://proxy.example/v1/audio/ berd-call session --stt-backend openai --stt-url wss://proxy.example/v1/realtime?intent=transcription ``` -In an Expert–Spokesperson call, the Expert reaches you through the Spokesperson: `berd-call message-spokesperson --re CURSOR [--resolves HANDOFF_ID]... TEXT` has it tell you in its own voice. Like the directives of Berd's Expert, the message must acknowledge the latest input (the Spokesperson's own speech doesn't count), or it comes back `spoke:false` with what was missed, and a handoff it names settles once the Spokesperson has said it. `speak` is refused unless the call was started with `--allow-direct-agent-speech`. With it, the Expert can also speak for itself: berd-call plays each `speak` reply through the configured TTS backend, taking turns with the Spokesperson's audio, and your speech interrupts either of them. Afterwards the Spokesperson gets a silent note of what you heard, so it can follow up without repeating it. A reply has to acknowledge everything you've said, as `speak --re` always does. Such a call holds no handoff open: each still reaches the Expert as a `handoff` row, but nothing waits for an answer, reminds the Expert, or holds back renewing or recovering the Spokesperson's session, and there is nothing for `--resolves` to name. `--voice`, `--rate`, `status` and `settings --tts` describe and change the Expert's voice. The Spokesperson keeps its own voice, shown as `session.spokesperson`: choose it with `--spokesperson-voice NAME`, at `start` or live with `settings`, and the menu bar offers both voices. It speaks as close to the agents' `--rate` as it can (0.25–1.5×), so the menu bar's one Speech Rate sets both; `--spokesperson-rate` gives it a speed of its own. Saved settings keep the Spokesperson's voice for the next call whose replies have their own, but not `--allow-direct-agent-speech`, so pass that on each `start`. +In an Expert–Spokesperson call, the Expert reaches you through the Spokesperson: `berd-call message-spokesperson --re CURSOR [--resolves HANDOFF_ID]... TEXT` has it tell you in its own voice. Like the directives of Berd's Expert, the message must acknowledge the latest input (the Spokesperson's own speech doesn't count), or it comes back `spoke:false` with what was missed, and a handoff it names settles once the Spokesperson has said it. `speak` is refused unless the call was started with `--allow-direct-agent-speech`. With it, the Expert can also speak for itself: berd-call plays each `speak` reply through the configured TTS backend, taking turns with the Spokesperson's audio, and your speech interrupts either of them. Afterwards the Spokesperson gets a silent note of what you heard, so it can follow up without repeating it. A reply has to acknowledge everything you've said, as `speak --re` always does. Such a call holds no handoff open: each still reaches the Expert as a `handoff` row, but nothing waits for an answer, reminds the Expert, or holds back renewing or recovering the Spokesperson's session, and there is nothing for `--resolves` to name. `--voice`, `--rate`, `status` and `settings --tts` describe and change the Expert's voice. The Spokesperson keeps its own voice, shown as `session.spokesperson`: choose it with `--spokesperson-voice NAME`, at `start` or live with `settings`, and the menu bar offers both voices. It speaks as close to the agents' `--rate` as it can (0.25–1.5×), so the menu bar's one Speech Rate sets both; `--spokesperson-rate` gives it a speed of its own. Saved settings keep the Spokesperson's voice for the next call whose replies have their own, but not `--allow-direct-agent-speech`, so pass that on each `start`. `--spokesperson-options JSON` sets the rest of the Spokesperson's realtime session, the options Berd keeps under Advanced: for example `{"turnDetection":"semantic_vad","eagerness":"low","noiseReduction":"near_field"}`. It takes Berd's names and ranges, applies when the Spokesperson's session starts, and isn't saved, so pass it on each `start`. The three URL flags take full endpoints and override only their matching service for that call. Expert–Spokesperson uses `--realtime-url` for both input and output; `--stt-url` applies only to conventional mode, and `--tts-url` also applies to the Expert's own voice. Without an override, each service uses its OpenAI endpoint (or its existing environment override). `OPENAI_API_KEY` supplies the CLI credential; the desktop app stores keys separately by endpoint URL in Keychain. diff --git a/src-tauri/crates/berd-call/src/cli_help.rs b/src-tauri/crates/berd-call/src/cli_help.rs index 72ae66275..cfc13f2c9 100644 --- a/src-tauri/crates/berd-call/src/cli_help.rs +++ b/src-tauri/crates/berd-call/src/cli_help.rs @@ -151,6 +151,7 @@ Each form also accepts: [--stt-backend macos|parakeet|openai] [--stt-model-dir PATH] [--mode conventional|expert-spokesperson] [--allow-direct-agent-speech] [--spokesperson-voice NAME] [--spokesperson-rate 0.25..1.5] + [--spokesperson-options JSON] [--realtime-url WS_URL] [--stt-url WS_URL] [--tts-url HTTP_URL] The host owns microphone capture, audio playback, transcript delivery, and @@ -162,7 +163,13 @@ In expert-spokesperson mode, the Expert speaks through the Spokesperson. --allow-direct-agent-speech also lets it speak to the user itself, through the configured TTS backend, in its own voice beside the Spokesperson's. --spokesperson-voice then chooses the Spokesperson's own voice. It speaks as close to --rate as it can, unless ---spokesperson-rate chooses its own speed."#; +--spokesperson-rate chooses its own speed. --spokesperson-options sets the rest of its +realtime session, as Berd's Advanced settings do, with a JSON object of model, +transcriptionModel, transcriptionLanguage, transcriptionPrompt, turnDetection +(server_vad|semantic_vad), eagerness, interruptResponse, createResponse, +vadThreshold, prefixPaddingMs, silenceDurationMs, idleTimeoutMs, noiseReduction +(off|near_field|far_field), reasoningEffort and maxOutputTokens. Saved settings +don't keep it, so pass it on each start."#; const SYNTHESIZE_HELP: &str = r#"Render text through a configured TTS backend into a new WAV file. diff --git a/src-tauri/crates/berd-call/src/main.rs b/src-tauri/crates/berd-call/src/main.rs index 94d8f26f5..2d3d45bbd 100644 --- a/src-tauri/crates/berd-call/src/main.rs +++ b/src-tauri/crates/berd-call/src/main.rs @@ -205,6 +205,9 @@ struct SessionConfig { /// The Spokesperson's own voice and speed when replies have theirs. spokesperson_voice: Option, spokesperson_rate: Option, + /// The Spokesperson's session options beyond its voice and speed. + spokesperson_options: + Option, } #[derive(Clone, Debug, Default, PartialEq)] @@ -3369,6 +3372,16 @@ fn apply_spokesperson_startup_settings( let (TtsBackendConfig::OpenAi { rate } | TtsBackendConfig::Siri { rate, .. } | TtsBackendConfig::Pocket { rate, .. }) = &session.tts; + if let Some(options) = &session.spokesperson_options { + let defaults = std::mem::replace(&mut spokesperson.session, options.clone()); + let chosen = &mut spokesperson.session; + chosen.model = chosen.model.take().or(defaults.model); + chosen.transcription_model = chosen + .transcription_model + .take() + .or(defaults.transcription_model); + (chosen.voice, chosen.speed) = (defaults.voice, defaults.speed); + } if let Some(voice) = &session.spokesperson_voice { spokesperson.session.voice = Some(voice.clone()); } @@ -6065,6 +6078,7 @@ fn parse_args(args: &[String]) -> Result { let mut expert_tts = false; let mut spokesperson_voice = None; let mut spokesperson_rate = None; + let mut spokesperson_options = None; let mut index = 2; while index < args.len() { let flag = args[index].as_str(); @@ -6113,6 +6127,11 @@ fn parse_args(args: &[String]) -> Result { .map_err(|_| "--spokesperson-rate must be a number".to_string())?, ) } + "--spokesperson-options" => { + spokesperson_options = Some( + berd_call::openai_realtime_protocol::RealtimeSpokespersonSessionOptions::from_call_json(value)?, + ) + } "--pcm-output-fd" => {} _ => return Err(format!("unknown argument: {flag}").into()), } @@ -6160,6 +6179,7 @@ fn parse_args(args: &[String]) -> Result { expert_tts, spokesperson_voice, spokesperson_rate, + spokesperson_options, }) } @@ -10751,6 +10771,7 @@ mod tests { expert_tts: false, spokesperson_voice: None, spokesperson_rate: None, + spokesperson_options: None, } ); @@ -10764,6 +10785,7 @@ mod tests { expert_tts: false, spokesperson_voice: None, spokesperson_rate: None, + spokesperson_options: None, } ); } @@ -10843,6 +10865,7 @@ mod tests { expert_tts: false, spokesperson_voice: None, spokesperson_rate: None, + spokesperson_options: None, } ); assert!(parse_args(&args(&[ @@ -10901,6 +10924,7 @@ mod tests { expert_tts: false, spokesperson_voice: None, spokesperson_rate: None, + spokesperson_options: None, }; let mut realtime = OpenAiSpokespersonConfig { endpoint: "ws://localhost".into(), @@ -10921,6 +10945,68 @@ mod tests { assert_eq!(realtime.voice(), "marin"); } + #[test] + fn spokesperson_options_reach_its_session_without_being_saved() { + let options = r#"{"model":"gpt-realtime-2","turnDetection":"semantic_vad","eagerness":"low","noiseReduction":"near_field","maxOutputTokens":512,"transcriptionPrompt":"Berd, Buzz"}"#; + let parse = |options: &str| { + parse_args(&args(&[ + "berd-call", + "session", + "--mode", + "expert-spokesperson", + "--tts-backend", + "openai", + "--spokesperson-options", + options, + ])) + }; + let mut realtime = OpenAiSpokespersonConfig { + endpoint: "ws://localhost".into(), + api_key: "test-key".into(), + session: berd_call::openai_realtime_protocol::RealtimeSpokespersonSessionOptions { + model: Some("env-model".into()), + transcription_model: Some("env-transcription".into()), + voice: Some("marin".into()), + speed: Some(1.0), + ..Default::default() + }, + semantic_transcript: Vec::new(), + }; + apply_spokesperson_startup_settings(&parse(options).unwrap(), &mut realtime); + let update = + berd_call::openai_realtime_protocol::spokesperson_session_update(&realtime.session); + let input = &update["session"]["audio"]["input"]; + assert_eq!(realtime.model(), "gpt-realtime-2"); + assert_eq!(input["turn_detection"]["type"], "semantic_vad"); + assert_eq!(input["turn_detection"]["eagerness"], "low"); + assert_eq!(input["noise_reduction"]["type"], "near_field"); + assert_eq!(input["transcription"]["model"], "env-transcription"); + assert_eq!(input["transcription"]["prompt"], "Berd, Buzz"); + assert_eq!(update["session"]["max_output_tokens"], 512); + assert_eq!(update["session"]["audio"]["output"]["voice"], "marin"); + for (options, error) in [ + (r#"{"voice":"cedar"}"#, "voice"), + (r#"{"speed":1.2}"#, "speed"), + (r#"{"vadThreshold":2}"#, "vadThreshold"), + (r#"{"silenceDurationMs":50}"#, "silenceDurationMs"), + (r#"{"maxOutputTokens":0}"#, "maxOutputTokens"), + (r#"{"model":" "}"#, "model"), + (r#"{"turnDetection":"loud"}"#, "--spokesperson-options"), + (r#"{"temperature":1}"#, "temperature"), + ("[]", "--spokesperson-options"), + ] { + assert!(parse(options).unwrap_err().contains(error), "{options}"); + } + let arguments = args(&["--spokesperson-options", options, "--voice", "Aaron"]); + assert_eq!( + saved_settings::persistable_arguments(&arguments), + args(&["--voice", "Aaron"]) + ); + // Turning the Spokesperson off and on keeps its options for later. + assert!(saved_settings::with_spokesperson(&arguments, false, false) + .contains(&"--spokesperson-options".to_string())); + } + #[test] fn expert_voice_is_an_expert_spokesperson_choice_with_its_own_tts() { let parse = |extra: &[&str]| { @@ -11123,6 +11209,7 @@ mod tests { expert_tts: false, spokesperson_voice: None, spokesperson_rate: None, + spokesperson_options: None, } ); assert!(parse_args(&args(&[ @@ -11187,6 +11274,7 @@ mod tests { expert_tts: false, spokesperson_voice: None, spokesperson_rate: None, + spokesperson_options: None, } ); assert!(parse_args(&args(&[ @@ -11634,6 +11722,7 @@ mod tests { expert_tts: false, spokesperson_voice: None, spokesperson_rate: None, + spokesperson_options: None, } ); } diff --git a/src-tauri/crates/berd-call/src/openai_realtime_protocol.rs b/src-tauri/crates/berd-call/src/openai_realtime_protocol.rs index 7a68cef77..5f58bb7f0 100644 --- a/src-tauri/crates/berd-call/src/openai_realtime_protocol.rs +++ b/src-tauri/crates/berd-call/src/openai_realtime_protocol.rs @@ -23,7 +23,7 @@ pub const OPENAI_REALTIME_VOICE_IDS: &[&str] = &[ static SPOKESPERSON_INSTRUCTIONS: LazyLock = LazyLock::new(|| realtime_role_instructions("Spokesperson")); -#[derive(Clone, Copy, Debug, Default, Deserialize, Serialize)] +#[derive(Clone, Copy, Debug, Default, Deserialize, Serialize, PartialEq)] #[serde(rename_all = "snake_case")] pub enum RealtimeTurnDetection { #[default] @@ -31,7 +31,7 @@ pub enum RealtimeTurnDetection { SemanticVad, } -#[derive(Clone, Copy, Debug, Default, Deserialize, Serialize)] +#[derive(Clone, Copy, Debug, Default, Deserialize, Serialize, PartialEq)] #[serde(rename_all = "snake_case")] pub enum RealtimeEagerness { Low, @@ -41,7 +41,7 @@ pub enum RealtimeEagerness { Auto, } -#[derive(Clone, Copy, Debug, Deserialize, Serialize)] +#[derive(Clone, Copy, Debug, Deserialize, Serialize, PartialEq)] #[serde(rename_all = "snake_case")] pub enum RealtimeNoiseReduction { Off, @@ -49,7 +49,7 @@ pub enum RealtimeNoiseReduction { FarField, } -#[derive(Clone, Copy, Debug, Deserialize, Serialize)] +#[derive(Clone, Copy, Debug, Deserialize, Serialize, PartialEq)] #[serde(rename_all = "snake_case")] pub enum RealtimeReasoningEffort { Default, @@ -59,7 +59,7 @@ pub enum RealtimeReasoningEffort { High, } -#[derive(Clone, Debug, Default, Deserialize, Serialize)] +#[derive(Clone, Debug, Default, Deserialize, PartialEq, Serialize)] #[serde(rename_all = "camelCase")] pub struct RealtimeSpokespersonSessionOptions { pub model: Option, @@ -81,6 +81,79 @@ pub struct RealtimeSpokespersonSessionOptions { pub max_output_tokens: Option, } +impl RealtimeSpokespersonSessionOptions { + /// A call's own options for its Spokesperson, as Berd's camelCase JSON within Berd's ranges. + /// The voice and speed have settings of their own, so they are refused here. + pub fn from_call_json(json: &str) -> Result { + let invalid = |error: String| format!("--spokesperson-options is invalid: {error}"); + let fields: serde_json::Map = + serde_json::from_str(json).map_err(|error| invalid(error.to_string()))?; + if let Some(field) = fields.keys().find(|field| { + !matches!( + field.as_str(), + "model" + | "transcriptionModel" + | "transcriptionLanguage" + | "transcriptionPrompt" + | "turnDetection" + | "eagerness" + | "interruptResponse" + | "createResponse" + | "vadThreshold" + | "prefixPaddingMs" + | "silenceDurationMs" + | "idleTimeoutMs" + | "noiseReduction" + | "reasoningEffort" + | "maxOutputTokens" + ) + }) { + return Err(invalid(format!("{field} is not a Spokesperson option"))); + } + let options: Self = serde_json::from_value(Value::Object(fields)) + .map_err(|error| invalid(error.to_string()))?; + let blank = + |value: &Option| value.as_ref().is_some_and(|value| value.trim().is_empty()); + let outside = |value: Option, range: std::ops::RangeInclusive| { + value.is_some_and(|value| !range.contains(&value)) + }; + for (name, bad) in [ + ("model must not be blank", blank(&options.model)), + ( + "transcriptionModel must not be blank", + blank(&options.transcription_model), + ), + ( + "vadThreshold must be between 0 and 1", + options + .vad_threshold + .is_some_and(|value| !(0.0..=1.0).contains(&value)), + ), + ( + "prefixPaddingMs must be between 0 and 2000", + outside(options.prefix_padding_ms, 0..=2_000), + ), + ( + "silenceDurationMs must be between 100 and 3000", + outside(options.silence_duration_ms, 100..=3_000), + ), + ( + "idleTimeoutMs must be between 1000 and 120000", + outside(options.idle_timeout_ms, 1_000..=120_000), + ), + ( + "maxOutputTokens must be between 1 and 4096", + outside(options.max_output_tokens, 1..=4_096), + ), + ] { + if bad { + return Err(invalid(name.into())); + } + } + Ok(options) + } +} + pub fn realtime_role_instructions(role: &str) -> String { let normalized = PROMPT_DOCUMENT.replace("\r\n", "\n"); let normalized = normalized.trim(); diff --git a/src-tauri/crates/berd-call/src/saved_settings.rs b/src-tauri/crates/berd-call/src/saved_settings.rs index f770da7a1..f8cfecd60 100644 --- a/src-tauri/crates/berd-call/src/saved_settings.rs +++ b/src-tauri/crates/berd-call/src/saved_settings.rs @@ -141,7 +141,8 @@ pub(crate) fn persistable_arguments(arguments: &[String]) -> Vec { .filter(|pair| { !matches!( pair[0].as_str(), - "--realtime-url" | "--stt-url" | "--tts-url" + // The caller keeps the Spokesperson's options and passes them each call. + "--realtime-url" | "--stt-url" | "--tts-url" | "--spokesperson-options" ) }) .flatten()