Skip to content

Commit efdd342

Browse files
authored
Merge pull request #583 from sysprog21/cleanup
Remove the audio latency probe
2 parents e87cc4d + 7943654 commit efdd342

11 files changed

Lines changed: 2 additions & 1532 deletions

File tree

‎.claude/skills/codetrial-rust/SKILL.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
---
22
name: codetrial-rust
3-
description: How Rust is written and reviewed in CodeTrial - which rules the build enforces (the [lints] table in Cargo.toml and scripts/check-rust-style.py) and which it leaves to judgment, how to suppress a lint, why a lint is switched off on purpose, the file-size ceilings, and the ownership, error, cast and test idioms the tree follows. Use when writing, refactoring or reviewing Rust under src/, tests/, examples/ or build.rs, when clippy or the rust-style gate fails, when tempted to write #[allow], a bare unwrap, an `as` cast, a glob import or a bool parameter, when a file nears its line ceiling, or when asked to audit or tighten the Rust style. Comments, commit messages and what stays out of a public issue are codetrial-conventions; running the gate is codetrial-verify.
3+
description: How Rust is written and reviewed in CodeTrial - which rules the build enforces (the [lints] table in Cargo.toml and scripts/check-rust-style.py) and which it leaves to judgment, how to suppress a lint, why a lint is switched off on purpose, the file-size ceilings, and the ownership, error, cast and test idioms the tree follows. Use when writing, refactoring or reviewing Rust under src/, tests/ or build.rs, when clippy or the rust-style gate fails, when tempted to write #[allow], a bare unwrap, an `as` cast, a glob import or a bool parameter, when a file nears its line ceiling, or when asked to audit or tighten the Rust style. Comments, commit messages and what stays out of a public issue are codetrial-conventions; running the gate is codetrial-verify.
44
---
55

66
# Writing Rust in CodeTrial

‎Cargo.toml‎

Lines changed: 0 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -138,12 +138,6 @@ tokio = { version = "1.54.1", features = ["test-util"] }
138138
# line; the SDK already depends on this crate.
139139
log = "0.4.34"
140140

141-
# The audio probe matches onsets to pulses, and `cargo test` builds an
142-
# example's tests without running them unless asked to.
143-
[[example]]
144-
name = "audio-latency"
145-
test = true
146-
147141
# scripts/hook-check holds every `cargo build` to the git hooks being in
148142
# effect; its build.rs says why it is a member of its own. `vendor/` holds path
149143
# dependencies, not members, and `desktop/` is a crate of its own with its own

‎README.md‎

Lines changed: 0 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -431,7 +431,6 @@ web/ Static browser application
431431
problem-bank/ Problem and judge source of truth
432432
config/ Env file templates and per-provider credentials
433433
docs/ Contracts and operational notes
434-
examples/ Standalone measurement probes built against the crate
435434
scripts/ Build, verification, and integration checks
436435
vendor/ Patched third-party crates
437436
tests/ Rust and browser tests

‎docs/latency-observability.md‎

Lines changed: 0 additions & 107 deletions
Original file line numberDiff line numberDiff line change
@@ -785,110 +785,3 @@ export file was named differently. CodeTrial keeps the tool split and the
785785
residual, joins asynchronous records by identity rather than by neighboring
786786
lines, and refuses an empty analysis. It uses milestones measured on its own
787787
clock instead of estimating provider stages it cannot see.
788-
789-
## Synthetic LiveKit audio suite
790-
791-
`python3 scripts/measure-audio-latency.py --output /tmp/audio-local` builds the
792-
`audio-latency` example against this checkout's locked dependencies and repeats
793-
three trials of the 100 ms and 20 ms native source queues, alternating their
794-
order. Each trial contains one labelled warmup pulse and twenty measured
795-
pulses. `--queues`, `--iterations`, and `--pulses` select the experiment. Output
796-
must be a new directory, created only after the build succeeds; an earlier
797-
experiment is never overwritten.
798-
799-
The default `--mode local` connects a native source to a local track sink and
800-
needs no service. `--mode room` publishes to a fresh, uniquely named LiveKit
801-
room and measures the remote decoded PCM sink. Load `LIVEKIT_URL`,
802-
`LIVEKIT_API_KEY`, and `LIVEKIT_API_SECRET` into the environment first, using
803-
the selected private config. No Gemini requests, candidate media, or interview
804-
dispatch are involved. The room mode creates ordinary synthetic participants
805-
and consumes LiveKit connection minutes. On Linux x86_64 the runner uses
806-
`scripts/fetch-clang.sh` unless `CXX` already selects a compiler.
807-
808-
The probe sends 24 kHz mono PCM in ten-millisecond frames, delivered in 100 ms
809-
bursts. A 100 ms, 1 kHz tone follows 300 ms of silence in each one-second trial
810-
window, and a final silent second follows the last pulse. RMS hysteresis detects
811-
an onset after decode. Both participants share one process and one monotonic
812-
clock, so host clock synchronization is not an assumption.
813-
814-
An onset counts for a pulse only within the one-second delivery window after
815-
its send, which ends no later than the next pulse's send. An onset in a gap
816-
between the window's deadline and the next send is recorded as late. Without an
817-
embedded pulse identity, a delivery slower than the whole period can land in
818-
the next pulse's window and look fast there; the pulse it belongs to is then
819-
missing, and under a constant overrun the warmup is always one of them.
820-
Variable delays and losses can still alias individual samples; this suite does
821-
not prove pulse identity or measure latency beyond its matching window. Zero,
822-
multiple or late onsets make a sample unsuccessful, the warmup included. Raw
823-
records include the send offset, first tone-frame capture completion, detected
824-
delivery latency, and on-time and late onset counts. The probe writes them
825-
before leaving the room, so a capture timeout or a failed unpublish or close
826-
cannot discard what was measured; the file retains measurement errors and
827-
teardown errors separately, and any of them exits nonzero. A pending teardown
828-
marker is saved first, then updated after every unpublish and close has been
829-
attempted. Each cleanup operation has a five-second deadline; a failed save
830-
also cannot skip cleanup. The raw result uses schema version 3; earlier files
831-
without teardown status are rejected. This is an acoustic onset estimate with
832-
ten-millisecond frame resolution; codec ringing, loss concealment, scheduling
833-
and jitter buffering can affect detection. It is not sample-accurate waveform
834-
correlation.
835-
836-
`local` measures source buffering and sink delivery; `room` additionally
837-
includes encoding, the network and SFU, and decode/jitter buffering. Neither
838-
measures a browser's speaker output, microphone hardware, Gemini inference,
839-
or production worker backlog. Capture completion is reported separately: it
840-
can precede delivery. In the pinned native implementation the configured
841-
source queue is a backpressure threshold; capacity is twice that threshold.
842-
Changing it can move waiting into the application worker instead of making
843-
the listener hear a reply sooner. Repeated trials are essential: an arbitrary
844-
phase relative to the native ten-millisecond timer alone can change onset
845-
latency by nearly a frame.
846-
847-
Each experiment saves `manifest.json` with revision, whether tracked files
848-
differ from it, one digest over the manifest, lockfile, build script and every
849-
file under `src/` and `examples/`, the executable hash, the Rust and C++
850-
compiler versions, and host identity. Each trial saves raw JSON.
851-
`summary.text`, `summary.json`, and `summary.csv` pool the measured samples by
852-
mode and queue size, retaining attempted, delivered, missing and late counts
853-
alongside mean, p50, p95, p99 and maximum for capture and delivery. Warmup is
854-
excluded from the statistics, and a missing warmup is counted separately.
855-
Percentiles interpolate ranks; fewer than twenty deliveries is labelled
856-
`small_n`. These are descriptive sample statistics, not confidence intervals or
857-
evidence that the settings caused a difference. Keep endpoint, host, workload,
858-
binary and network conditions matched when comparing experiments, and inspect
859-
individual trial files before pooling.
860-
861-
The report tool also serves as an acceptance check:
862-
863-
```sh
864-
python3 scripts/audio-latency-report.py /tmp/audio-local/iter-*.json
865-
python3 scripts/audio-latency-report.py /tmp/audio-local/iter-*.json \
866-
--format json --max-p95-ms 50 --max-missing-rate 0
867-
```
868-
869-
Thresholds are experiment-specific and compare unrounded statistics; only the
870-
printed values are rounded. The default refuses any unsuccessful sample. A
871-
report with no delivered samples, or with a missing warmup, always fails,
872-
including when a nonzero missing-rate budget is allowed. Invalid data, an empty
873-
or interrupted run, a trial that left no result, failed or incomplete teardown,
874-
invalid timing windows, or the same run given twice fails rather than producing
875-
a healthy verdict; the runner then writes the refusal reason in place of
876-
statistics, as a JSON error object and a one-column CSV in the machine-readable
877-
summaries. The ordinary gate checks the report's validation, statistics, export
878-
formats and budget exit statuses; real-room experiments are separate from the
879-
credential-free gate.
880-
881-
### Lessons from lk-latency
882-
883-
The suite adapts the methodology of
884-
[lk-latency](https://github.com/haffi96/lk-latency), a LiveKit video latency
885-
harness: fresh rooms per iteration, raw timing archives, build identity and
886-
pooled statistics. It keeps CodeTrial's released SDK rather than the
887-
separately pinned revision that harness needs for per-stage hooks, so it does
888-
not invent encoder, network or decode timestamps it cannot observe. Several
889-
rules carry over. A run that never delivers media cannot pass. Tracks are
890-
unpublished before the room closes, so shutdown negotiation does not look like
891-
a media failure. Credentials reach the probe through the environment, never
892-
argv. Every timing belongs to an explicit pulse, and an ambiguous match is
893-
rejected rather than resolved in favor of the fastest. The endpoint is named
894-
for what it is, decoded PCM at the sink, with no claim about audible output.

0 commit comments

Comments
 (0)