Skip to content

cascade simulator on live #240

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
raghavm243512 wants to merge 12 commits into
base: main
Choose a base branch
from
+2,870 −1,123
Open

cascade simulator on live #240

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
12 commits
Select commit Hold shift + click to select a range
36a2deb
cascade simulator on live
raghavm243512 Oct 19, 2026
15aecc6
Apply pre-commit
raghavm243512 Oct 19, 2026
31db1e5
voice control for cascade
raghavm243512 Oct 19, 2026
844384e
Remove unused backend and role scaffolding
tara-bogavelli Sep 16, 2026
2c5c68e
address comments
raghavm243512 Oct 20, 2026
9116269
bug fixes and optional silence
raghavm243512 Oct 22, 2026
85b2cf2
concurrency fix
raghavm243512 Oct 23, 2026
1d2230b
error case
raghavm243512 Oct 23, 2026
5f2e8e7
remove user spec and fix preflight
raghavm243512 Oct 23, 2026
1ba4131
Fix comment
JosephMarinier Oct 23, 2026
d3c301c
Set one semaphore per model
JosephMarinier Oct 23, 2026
c6c04a2
update config
raghavm243512 Oct 24, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
43 changes: 30 additions & 13 deletions .env.example
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -175,8 +175,7 @@ EVA_MODEL_LIST='[
"model_name": "gpt-5.2",
"litellm_params": {
"model": "openai/gpt-5.2",
"api_key": "os.environ/OPENAI_API_KEY",
"max_parallel_requests": 5
"api_key": "os.environ/OPENAI_API_KEY"
},
"model_info": {"base_model": "gpt-5.2"}
},
Expand All @@ -186,17 +185,15 @@ EVA_MODEL_LIST='[
"model": "vertex_ai/gemini-3.1-pro-preview",
"vertex_project": "your-gcp-project-id",
"vertex_location": "global",
"vertex_credentials": "os.environ/GOOGLE_APPLICATION_CREDENTIALS",
"max_parallel_requests": 5
"vertex_credentials": "os.environ/GOOGLE_APPLICATION_CREDENTIALS"
}
},
{
"model_name": "us.anthropic.claude-opus-4-6",
"litellm_params": {
"model": "bedrock/us.anthropic.claude-opus-4-6-v1",
"aws_access_key_id": "os.environ/AWS_ACCESS_KEY_ID",
"aws_secret_access_key": "os.environ/AWS_SECRET_ACCESS_KEY",
"max_parallel_requests": 5
"aws_secret_access_key": "os.environ/AWS_SECRET_ACCESS_KEY"
}
}
]'
Expand All @@ -215,6 +212,11 @@ EVA_MODEL_LIST='[
#r 1,100,1
#v EVA_MAX_CONCURRENT_CONVERSATIONS=1

#i Maximum number of concurrent judge LLM calls (metrics and validation) to each judge model.
#d int
#r 1,100,1
#v EVA_MAX_CONCURRENT_JUDGES=3

#i Conversation time limit in seconds.
#d int
#r 30,10000,10
Expand Down Expand Up @@ -303,24 +305,35 @@ EVA_MODEL_LIST='[
# --- User simulator provider ---
#i Caller provider. ElevenLabs is the backward-compatible default.
#d enum
#e elevenlabs,openai_realtime
#e elevenlabs,openai_realtime,cascade
EVA_USER_SIMULATOR__PROVIDER=elevenlabs

#i OpenAI Realtime caller model. Used only when provider=openai_realtime.
#i Caller model. For openai_realtime, the Realtime model (default gpt-realtime-1.5). For cascade, the caller LLM
#i deployment name from EVA_MODEL_LIST (required).
#d string
#x EVA_USER_SIMULATOR__PROVIDER=openai_realtime
#x EVA_USER_SIMULATOR__PROVIDER=openai_realtime,cascade
#v EVA_USER_SIMULATOR__MODEL=gpt-realtime-1.5

#i OpenAI voice for the female caller persona.
#i Voice for the female caller persona. Voice name or ID depending on provider.
#d string
#x EVA_USER_SIMULATOR__PROVIDER=openai_realtime
#x EVA_USER_SIMULATOR__PROVIDER=openai_realtime,cascade
#v EVA_USER_SIMULATOR__FEMALE_VOICE=marin

#i OpenAI voice for the male caller persona.
#i Voice for the male caller persona. Voice name or ID depending on provider.
#d string
#x EVA_USER_SIMULATOR__PROVIDER=openai_realtime
#x EVA_USER_SIMULATOR__PROVIDER=openai_realtime,cascade
#v EVA_USER_SIMULATOR__MALE_VOICE=cedar

#i Cascade caller streaming STT model (ElevenLabs Scribe realtime). Requires ELEVENLABS_API_KEY.
#d string
#x EVA_USER_SIMULATOR__PROVIDER=cascade
#v EVA_USER_SIMULATOR__STT_MODEL=scribe_v2_realtime

#i Cascade caller Cartesia TTS model. Requires CARTESIA_API_KEY.
#d string
#x EVA_USER_SIMULATOR__PROVIDER=cascade
#v EVA_USER_SIMULATOR__TTS_MODEL=sonic-3.5

# --- Language (mutually exclusive with Accent and Behavior) ---
#i ISO 639-1 language code for the user simulator. Datasets must exist for the selected language. Pattern for the agent ID pairs below: EVA_{LANG}_USER_{F|M}.
#d enum
Expand Down Expand Up @@ -588,3 +601,7 @@ EVA_EN_USER_M=your_elevenlabs_agent_id_for_default_user_m
#d enum
#e DEBUG,INFO,WARNING,ERROR,CRITICAL
#v EVA_LOG_LEVEL=INFO

#i Enable preflight checks before running benchmarks.
#d bool
#v EVA_PREFLIGHT=false
30 changes: 10 additions & 20 deletions docs/llm_configuration.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -154,17 +154,15 @@ EVA_MODEL_LIST='[
"litellm_params": {
"model": "azure/gpt-5.2",
"api_key": "os.environ/AZURE_KEY_A",
"api_base": "https://endpoint-a.openai.azure.com",
"max_parallel_requests": 5
"api_base": "https://endpoint-a.openai.azure.com"
}
},
{
"model_name": "gpt-5.2",
"litellm_params": {
"model": "azure/gpt-5.2",
"api_key": "os.environ/AZURE_KEY_B",
"api_base": "https://endpoint-b.openai.azure.com",
"max_parallel_requests": 5
"api_base": "https://endpoint-b.openai.azure.com"
}
}
]'
Expand Down Expand Up @@ -194,23 +192,15 @@ EVA_MODEL_LIST='[

## Concurrency Control

Set `max_parallel_requests` per deployment to limit concurrent API calls:
EVA bounds load with two settings rather than per-deployment limits:

```bash
EVA_MODEL_LIST='[
{
"model_name": "gpt-5.2",
"litellm_params": {
"model": "azure/gpt-5.2",
"api_key": "os.environ/AZURE_API_KEY",
"api_base": "https://your-resource.openai.azure.com",
"max_parallel_requests": 5
}
}
]'
```
- `EVA_MAX_CONCURRENT_CONVERSATIONS`: conversations running at once (assistant and user simulator calls).
- `EVA_MAX_CONCURRENT_JUDGES`: judge LLM calls in flight at once to each judge model, across all metrics and

@JosephMarinier JosephMarinier Oct 23, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For the record, I think this isn't quite "all metrics": when Gemini reports "no audio", speech fidelity falls back to the File Upload API and calls genai.Client directly (src/eva/metrics/speech_fidelity_base.py), so those calls skip this limit. They also skipped the router's default_max_parallel_requests on main, so it isn't a regression. Let's ignore it for now; I'm just noting it in case someone hits rate limits there later.

validation. Each model has its own limit, so judges on different models don't wait on each other.

You can also set `rpm` (requests per minute) and `tpm` (tokens per minute) limits.
At most `EVA_MAX_CONCURRENT_JUDGES` × the number of judge models are in flight at once. Do not set `max_parallel_requests` on deployments: LiteLLM
rejects calls over that cap with an immediate 429 instead of queueing them. `rpm`/`tpm` are also turned into a
per-deployment parallel cap by LiteLLM, so keep them well above the limits above if you set them.

## Advanced: Model Parameters

Expand All @@ -228,7 +218,7 @@ Configure model parameters (temperature, max_tokens, reasoning_effort, top_p, fr
- See LiteLLM docs: https://docs.litellm.ai/docs/providers

### "Rate Limit Exceeded"
- Adjust `max_parallel_requests` per deployment in `EVA_MODEL_LIST`
- Lower `EVA_MAX_CONCURRENT_JUDGES` (metrics) or `EVA_MAX_CONCURRENT_CONVERSATIONS` (conversations)

## Retry Logic and Rate Limits

Expand Down
5 changes: 2 additions & 3 deletions pyproject.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -25,15 +25,14 @@ dependencies = [
"elevenlabs>=2.53.0",
"openai>=2.36.0",
"anthropic>=0.83.0",
"litellm==1.85.0",
"litellm==1.105.1",
"deepgram-sdk>=7",
"onnxruntime>=1.16.0",
"aioboto3>=12.0.0",
"google-genai>=1.69.0",
"google-cloud-speech>=2.0.0",
"google-cloud-texttospeech>=2.0.0",
"azure-cognitiveservices-speech>=1.31.0",
"cartesia>=1.0.0",
"cartesia>=3.0.0",
"assemblyai>=0.17.0",
"setuptools>=65.0.0",
"fastapi>=0.100.0",
Expand Down
Loading
Loading