Add cascade interruptions and out-of-turn behaviors

configs/caller_phrases.yaml

@@ -0,0 +1,69 @@
+de:
+  backchannels:
+  - mhm
+  - aha
+  - ja
+  barge_in_openers:
+  - Moment—
+  - Entschuldigung—
+  - Warte—
+  - Also—
+en:
+  backchannels:
+  - uh-huh
+  - mm-hmm
+  barge_in_openers:
+  - Wait—
+  - Sorry—
+  - Hold on—
+  - Actually—
+es:
+  backchannels:
+  - ajá
+  - mmm
+  - ya
+  barge_in_openers:
+  - Espera
+  - Perdona
+  - Un momento
+  - Bueno
+fr:
+  backchannels:
+  - hum hum
+  - oui
+  - d’accord
+  barge_in_openers:
+  - Attendez
+  - Pardon
+  - Juste
+  - En fait
+fr-CA:
+  backchannels:
+  - hum hum
+  - ouais
+  - OK
+  barge_in_openers:
+  - Attends
+  - Excuse
+  - Juste
+  - En fait
+hi:
+  backchannels:
+  - हूँ
+  - अच्छा
+  - हाँ
+  barge_in_openers:
+  - रुकिए
+  - सुनिए
+  - माफ़ कीजिए
+  - असल में
+ko:
+  backchannels:
+  - 네
+  - 음
+  - 아
+  barge_in_openers:
+  - 잠깐만요
+  - 죄송한데
+  - 아니요
+  - 근데

configs/prompts/simulation.yaml

@@ -507,3 +507,146 @@ user_simulator:
     For other languages, spell out characters using the standard phonetic alphabet or character names used in that language.
 
     IMPORTANT: Before ending the conversation, confirm with the agent that there are no outstanding actions. The end_call tool should only be called in a turn that is a brief goodbye — never in the same turn where you are providing the agent with data, an identifier, a request to transfer to a live agent, an approval to proceed, or any kind of additional information.
+
+  interruption_decision: |
+    You are analyzing a conversation to decide if the user should interrupt the agent.
+
+    This is what the user called to accomplish, so you can judge what they still have left
+    to do. The fields mean:
+
+    - GOAL: what the user is trying to achieve overall.
+    - MUST HAVE: non-negotiable requirements. The user will never accept an outcome that
+      fails any of these, and will not hang up until all of them are met.
+    - NICE TO HAVE: things the user wants but will give up if necessary.
+    - HOW THEY EVALUATE OPTIONS: the steps the user follows when the agent presents choices.
+    - RESOLVED WHEN: the success condition. Once this is met the user says a brief goodbye
+      and ends the call, so there is nothing left worth interrupting for.
+    - FAILED WHEN: the failure condition. This also ends the call.
+    - ESCALATION: how the user handles being transferred to a live agent.
+
+    <user_goal>
+    {user_goal}
+    </user_goal>
+
+    Conversation history (most recent at bottom):
+
+    <conversation_history>
+    {conversation_history}
+    </conversation_history>
+
+    The agent is CURRENTLY speaking (you can see their ongoing speech in the conversation above).
+
+    Based on the conversation so far, should the user interrupt the agent NOW?
+
+    Consider:
+    - Has the user heard enough to understand what the agent is asking or saying?
+    - Has the user heard enough to have a response, question, or correction ready?
+    - Did the agent just complete the sentence which has all the pertinent information the user was looking for?
+    - Do NOT repeatedly interrupt the agent if it has spoken only a few words (roughly fewer than 5 words in English, or the equivalent short fragment in whatever language the conversation is in — word counts are not comparable across languages).
+    - Are the user's request(s) basically accomplished and the user is likely to hang up on the next turn (if so it should not interrupt)?
+    - Is this a logical point in the conversation to interrupt?
+
+    Respond with ONLY "YES" if the user should interrupt now, or "NO" if they should keep listening.
+
+  backchannel_decision: |
+    You simulate a natural listener who occasionally makes a brief continuer sound to show they're following along.
+
+    The conversation may be in any language. Judge the FUNCTION of what is said, not the
+    presence of any particular word: every language has these sounds, and the caller draws
+    its own from a vocabulary configured for the conversation's language. The English
+    examples below illustrate the judgement, not the words to look for.
+
+    <conversation_history>
+    {conversation_history}
+    </conversation_history>
+
+    The agent is still speaking [CURRENTLY SPEAKING, INCOMPLETE]. Ignore the trailing incomplete word/phrase — focus only on the COMPLETE sentences delivered so far in the agent's current turn.
+
+    Continuers (English: "uh-huh", "mm-hmm", "yeah"; every language has equivalents) are brief
+    sounds that mean "I'm listening, keep going." They:
+    - Happen naturally during extended speech
+    - Show engagement without interrupting
+    - Are NOT responses to specific content — just signals of attention
+
+    Say YES if:
+    - The agent has completed at least 2 full, substantive sentences in their current turn
+      (Short phrases like "Thanks for your patience" or "Let me check on that" don't count as substantive)
+    - The user hasn't spoken or backchanneled recently (check the last 3 exchanges for ANY brief
+      acknowledgement sound from the user, in whatever language the conversation is in)
+    - It would feel natural to briefly signal "I'm still here"
+
+    Say NO if:
+    - The agent just started speaking (fewer than 2 substantive sentences)
+    - The user spoke OR backchanneled within the last 2-3 exchanges
+    - The agent's current turn contains or ends with a question
+    - The agent is wrapping up or about to finish their thought
+
+    Frequency guidance:
+    - Continuers are occasional, not constant
+    - Even when conditions seem right, real listeners only backchannel sometimes
+    - Aim for roughly 1 continuer per 4-6 sentences of extended agent speech
+    - When in doubt, say NO — silence is also natural
+    - Too few continuers is better than too many
+
+    Examples (English, for illustration — apply the same judgement in any language):
+
+    AGENT: "Hi there! How can I hel [CURRENTLY SPEAKING, INCOMPLETE]"
+    → NO (just started)
+
+    AGENT: "Thanks for your patience. [CURRENTLY SPEAKING, INCOMPLETE]"
+    → NO (only 1 short sentence, not substantive enough)
+
+    AGENT: "Sure, I can help with that. First I'll need to verify your account. Could you provide your email or your name and zi [CURRENTLY SPEAKING, INCOMPLETE]"
+    → NO (agent is asking a question)
+
+    AGENT: "No problem. We can use your name and zip code instead. Let me look that up for you. I'll check our system now and see if I can fin [CURRENTLY SPEAKING, INCOMPLETE]"
+    → YES (3 substantive sentences, agent explaining process)
+
+    AGENT: "I found your order. It includes a keyboard, thermostat, and headphones. The order was delivered last Tuesday. Now for the exchange, we have a few opti [CURRENTLY SPEAKING, INCOMPLETE]"
+    → YES (extended explanation with specific details)
+
+    [If the user made a brief acknowledgement sound 2 exchanges ago]
+    AGENT: "...and those are the available options. Now I'll need your input on which [CURRENTLY SPEAKING, INCOMPLETE]"
+    → NO (user backchanneled recently, don't do it again so soon)
+
+    Respond with ONLY "YES" or "NO".
+
+  cascade_next_interruption: |
+    You just said this line to the agent:
+
+    <what_you_just_said>
+    {utterance}
+    </what_you_just_said>
+
+    Write the single line you would say to cut the agent off mid-reply, if its
+    answer turns out to need correcting or clarifying.
+
+    Rules:
+    - Write it as a natural interruption, not a full turn. Short.
+    - It must follow from your own goal, not from any specific thing the agent
+      might say — you have not heard the reply yet.
+    - Write it in the same language as the line above; the caller does not switch
+      languages mid-call.
+    - Reply with ONLY the spoken line, nothing else. No quotes, no labels.
+    - If you would have no reason to interrupt whatever the agent says next,
+      reply with exactly NONE.
+
+  cascade_relevance_gate: |
+    The caller prepared this line a moment ago, before hearing the rest of what
+    the agent is currently saying:
+
+    <prepared_line>
+    {candidate}
+    </prepared_line>
+
+    Here is what the agent has said so far in its current turn:
+
+    <heard_so_far>
+    {heard}
+    </heard_so_far>
+
+    Would saying the prepared line right now still make sense as a natural
+    interruption? Answer NO if it has become irrelevant, already been addressed,
+    or would read as a non-sequitur.
+
+    Respond with ONLY "YES" or "NO".

docs/assistant_server_contract.md

@@ -125,6 +125,7 @@ and decoding.
 | `start` | Stream opened, contains `streamSid` | Parse `msg["start"]["streamSid"]` |
 | `media` | Audio chunk (8 kHz mulaw, base64) | `parse_twilio_media_message(raw)` → `bytes` |
 | `stop` | Stream closed | Break receive loop |
+| `truncate` | Cascade tick-driven barge-in; discard audio after `audio_end_ms` | OpenAI Realtime truncates its active audio item |
 | `user_speech_start` | User began speaking (wall-clock timestamp) | `msg["timestamp_ms"]` |
 | `user_speech_stop` | User stopped speaking (wall-clock timestamp) | `msg["timestamp_ms"]` |
 

docs/cascade_user_simulator.md

@@ -57,3 +57,37 @@ The caller ends with `goodbye` when its LLM calls `end_call`, or `timeout` when
 conversation budget expires. Sustained assistant silence ends with
 `inactivity_timeout`. A stalled tick-driven provider ends with the distinct
 `provider_stalled` reason and preserves partial artifacts for diagnosis.
+
+## Experimental out-of-turn behaviors
+
+These behaviors are opt-in and are still being tested. Enable them independently
+in the cascade configuration:
+
+```dotenv
+EVA_USER_SIMULATOR__DECISION_LLM=user-llm
+EVA_USER_SIMULATOR__ENABLE_BACKCHANNEL=true
+EVA_USER_SIMULATOR__ENABLE_INTERRUPTIONS=true
+EVA_USER_SIMULATOR__SPECULATIVE_GENERATION=true
+```
+
+All three behavior flags default to `false`. `decision_llm` selects the deployment
+for listener and relevance checks; its default is `user-llm`.
+
+Backchannels play cached continuers while the assistant speaks without taking an
+ordinary caller turn. Interruptions generate goal-aware barge-ins, limited to one
+per detected assistant turn. Speculative generation pre-renders candidate
+interruptions and checks their relevance before use; it is useful when
+interruptions are also enabled.
+
+The tick-driven adapter reports the played audio position when a barge-in reaches
+the wire, discards unheard buffered audio, and requests OpenAI Realtime item
+truncation. The real-time adapter retains its normal streaming behavior.
+
+`configs/caller_phrases.yaml` supplies localized continuers and interruption
+openers. `scripts/add_culture_data.py` generates these phrase sets alongside other
+language data. Phrase audio is cached across conversations.
+
+The decision trace adds listener verdicts, skipped-check context, interruption
+outcomes, and speculative-candidate decisions. Use it alongside the event log and
+audio to evaluate actual timing; passing unit tests alone does not establish
+behavioral quality.

scripts/add_culture_data.py

@@ -103,6 +103,7 @@
 REPO_ROOT = Path(__file__).resolve().parent.parent
 DATA_DIR = REPO_ROOT / "data"
 INITIAL_MESSAGES_PATH = REPO_ROOT / "configs" / "agents" / "initial_messages.yaml"
+CALLER_PHRASES_PATH = REPO_ROOT / "configs" / "caller_phrases.yaml"
 WER_CONFIGS_DIR = REPO_ROOT / "src" / "eva" / "utils" / "wer_normalization" / "configs"
 
 DEFAULT_MODEL = "gpt-5.5-2026-04-23"
@@ -353,6 +354,63 @@ def _update_initial_messages(language: str, message: str) -> None:
     INITIAL_MESSAGES_PATH.write_text(yaml.safe_dump(existing, allow_unicode=True, sort_keys=True), encoding="utf-8")
 
 
+async def _translate_caller_phrases(
+    language: str, language_name: str, llm: LLMClient, overwrite: bool = False
+) -> dict[str, list[str]] | None:
+    """Generate the simulated caller's out-of-turn vocabulary for ``language``.
+
+    These are the continuers and barge-in openers the caller speaks *as audio*, so
+    they must be things a native speaker actually says, not translations of the
+    English ones — "uh-huh" has no word-for-word equivalent in most languages.
+
+    Returns None when the language already has phrases and ``overwrite`` is unset.
+    """
+    existing: dict[str, Any] = {}
+    if CALLER_PHRASES_PATH.exists():
+        existing = yaml.safe_load(CALLER_PHRASES_PATH.read_text(encoding="utf-8")) or {}
+    if language in existing and not overwrite:
+        return None
+
+    english = existing.get("en", {})
+    prompt = (
+        f"You are localising a simulated phone caller for {language_name}.\n\n"
+        "Produce two short lists of things the caller says out loud:\n\n"
+        "1. backchannels — brief continuer sounds meaning 'I am listening, keep going', "
+        "said while the other person is still talking. Give the sounds a native speaker "
+        "actually makes, not translations of English ones.\n"
+        "2. barge_in_openers — the very first word or two of an interruption, said just "
+        "before the interrupting sentence. Keep them to one or two words so they sound "
+        "like a real cut-in.\n\n"
+        "These are spoken aloud by a text-to-speech voice, so use ordinary spelling with "
+        "no stage directions, no parentheses and no transliteration hints. Give 2-3 "
+        "backchannels and 3-4 openers.\n\n"
+        f"For reference, the English set is: {json.dumps(english, ensure_ascii=False)}\n\n"
+        'Return JSON: {"backchannels": ["..."], "barge_in_openers": ["..."]}'
+    )
+    text, _ = await llm.generate_text(
+        [{"role": "user", "content": prompt}],
+        response_format={"type": "json_object"},
+    )
+    data = extract_and_load_json(text)
+    backchannels = [str(p).strip() for p in (data.get("backchannels") or []) if str(p).strip()]
+    openers = [str(p).strip() for p in (data.get("barge_in_openers") or []) if str(p).strip()]
+    if not backchannels or not openers:
+        raise ValueError(f"Caller phrase generation returned an incomplete result: {data!r}")
+    return {"backchannels": backchannels, "barge_in_openers": openers}
+
+
+def _update_caller_phrases(language: str, phrases: dict[str, list[str]]) -> None:
+    """Merge one language's phrase set into configs/caller_phrases.yaml."""
+    existing: dict[str, Any] = {}
+    if CALLER_PHRASES_PATH.exists():
+        existing = yaml.safe_load(CALLER_PHRASES_PATH.read_text(encoding="utf-8")) or {}
+    if existing.get(language) == phrases:
+        return
+    existing[language] = phrases
+    CALLER_PHRASES_PATH.parent.mkdir(parents=True, exist_ok=True)
+    CALLER_PHRASES_PATH.write_text(yaml.safe_dump(existing, allow_unicode=True, sort_keys=True), encoding="utf-8")
+
+
 async def _translate_aliases(
     name_to_base: dict[str, list[str]],
     language_name: str,
@@ -680,6 +738,16 @@ async def amain(args: argparse.Namespace) -> int:
         _update_initial_messages(args.language, initial_message)
         logger.info(f"Updated {INITIAL_MESSAGES_PATH}")
 
+    logger.info(f"Generating caller out-of-turn phrases for {args.language_name}")
+    caller_phrases = await _translate_caller_phrases(args.language, args.language_name, llm, args.overwrite_all)
+    if caller_phrases is None:
+        logger.info(f"Caller phrases for {args.language} already present — skipping")
+    elif args.dry_run:
+        logger.info(f"[dry-run] would write caller phrases for {args.language}: {caller_phrases}")
+    else:
+        _update_caller_phrases(args.language, caller_phrases)
+        logger.info(f"Updated {CALLER_PHRASES_PATH}: {caller_phrases}")
+
     for domain in domains:
         logger.info(f"=== Domain: {domain} ===")
         if domain == "airline":

src/eva/__init__.py

@@ -7,7 +7,7 @@
 
 # Bump simulation_version when changes affect benchmark outputs (agent code,
 # user simulator, orchestrator, simulation prompts, agent configs, tool mocks).
-simulation_version = "2.0.34"
+simulation_version = "2.0.35"
 
 # Bump metrics_version when changes affect metric computation (metrics code,
 # judge prompts, pricing tables, postprocessor).