[raghavm243512]

Live version of #214

Not as big as it looks, half the lines are from the measurement script which doesn't interact with anything

key changes vs main:

• .env.example added a missing property, exact format of configurations is still open to change
• pyproject.toml version bumps needed for eventual support of openAI SDK 3, model maps were outdated and not working with gpt 6+
• scripts added experiment script for measuring timing, not used in evals in any way so can ignore for easier review. Removed fixed name from text_only
• llm.py new litellm conventions require using a semaphore because you cannot limit max_parallel requests in the same way. Instead of using its own semaphore, it now shows error 429. Keep old functionality (need to move this parameter though)
• config.py need to extend this for interruptions, for now basic user config. No flexibility yet, to add later
• preflight.py has some missing checks plus bug to fix for later
• user_simulator/cascade just the cascade simulator, with streaming and other latency optimization (resampling) so it is fast when we add interruption

Model intelligence does not need to be tested since the new plan is to use a large model as primary mind, and smaller one for interruptions only, and these will be the same tier of model we already use in ElevenLabs.

Other language setting and prompt flows to user sim and voice ID is controllable, so will work for other languages

You may wonder why I use httpx over cartesiaSDK. I didn't see a way to stream without httpx. There is also some odd sample rate stuff, this actually knocks 50ms or so off the STT lag / net transcript time. Not really sure why

Updated LiteLLM changes: new litellm doesn't support that innate secondary semaphore. We were actually using this incorrectly. New changes: separate knobs just so concurrency doesn't break pipelining. Rather than reintroduce this parameter, we drop it in favor of uniformly applied concurrency constraints (max conversations bounds everything in conversation, and separate judge knob bounds judges). The most a model can ever concur is the sum of these if the models are the same (and user LLM if that's being used for all 3 slots, but that's the user's problem to deal with)

Preflight fix: ElevenLabs TTS was breaking preflight, adjusted conditions to check connection is error free rather than wait for the speaking to conclude

[JosephMarinier]

Since we don't know our users' setups, I think the default value should be a conservative 1. A limit that's too high could lead to real issues, whereas a limit that's too low only risks being slower. And unless max_concurrent_conversations is very high (it's 1 by default), I think conversations are long enough for judge calls to occur in sequence within the time it takes to simulate the conversation.

Also, I know that's not new, but who are we to set a maximum of le=100? 😝 We can remove that.

Does that make sense?

Suggested change

	le=100,
	description="Maximum number of concurrent conversations",
	)
	max_concurrent_judges: int = Field(
	3,
	ge=1,
	le=100,
	description="Maximum number of concurrent conversations",
	)
	max_concurrent_judges: int = Field(
	1,
	ge=1,

[JosephMarinier]

For the record, I think this isn't quite "all metrics": when Gemini reports "no audio", speech fidelity falls back to the File Upload API and calls genai.Client directly (src/eva/metrics/speech_fidelity_base.py), so those calls skip this limit. They also skipped the router's default_max_parallel_requests on main, so it isn't a regression. Let's ignore it for now; I'm just noting it in case someone hits rate limits there later.