Skip to content

feat(full_duplex): add FD-Bench timing evaluation - #41

Closed
wangxiangruirui wants to merge 1 commit into
X-LANCE:mainfrom
wangxiangruirui:feat/full-duplex-fdbench-timing
Closed

wangxiangruirui wants to merge 1 commit into
X-LANCE:mainfrom
wangxiangruirui:feat/full-duplex-fdbench-timing

Conversation

@wangxiangruirui

Copy link
Copy Markdown
Contributor

Summary

Add a full_duplex task for scoring saved, shared-clock user/system speech
timelines with IRD, FSED, ERT, and EIT. Two reusable, CPU-only in-process nodes
validate the JSONL contract and calculate the four metrics in one traversal;
each atomic route selects one primary score while retaining all four results.

The independent implementation targets the fixed FD-Bench procedure at commit
8a4b7df1b4dcb0fc50a7a5660247a2ffe8d394eb. It preserves strict boundaries,
candidate overwrites, special IRD endpoints, positive-tick filtering,
per-event integer-ms truncation, and pooled medians. Empty pools retain the
compatibility 0.0 value with explicit no_events and measurement warnings.

Reports retain detailed arrays and the event ledger once, with compact node
traces. Optional validated measurement provenance records VAD/playback/reference
settings without changing scores; missing/partial provenance remains valid v1
input and produces a comparability warning. JSON and terminal summaries
distinguish empty pools from observed sub-millisecond events reported as 0 ms.

No model serving, VAD inference, API integration, upstream source, benchmark
recordings, weights, or new runtime dependencies are included. The fixtures are
synthetic, including 34 frozen cases generated from the hash-locked official
timing oracle; normal CI reads only their inputs and expected outputs.

PR Type

  • New task
  • New metric for an existing task
  • New pipeline route for an existing metric
  • Node/tool/version change for an existing route
  • Maintenance only

Guides followed: new task, new metric, node change, and contributing.

Evaluation Identity

  • Reported metrics and singleton execution selectors: ird, fsed, ert, eit.
  • New atomic IDs:
    • full_duplex.any.ird.duplex_timeline_v1.fdbench_timing_v1
    • full_duplex.any.fsed.duplex_timeline_v1.fdbench_timing_v1
    • full_duplex.any.ert.duplex_timeline_v1.fdbench_timing_v1
    • full_duplex.any.eit.duplex_timeline_v1.fdbench_timing_v1
  • New nodes: normalization/duplex_timeline@v1, scoring/fdbench_timing@v1.
  • Default route changed: no existing task defaults changed; new task defaults
    to IRD. FSED-filtered discovery retains the task's IRD default metadata.
  • Score comparability: new task/protocol only. ERT/EIT have no universal ranking
    direction. Empty-pool zero is not a measured zero latency. Playback and VAD
    provenance must be controlled before comparing saved recordings.

Changed Declarations

  • Task manifest/routes: added.
  • Node manifests and node_env.yaml: added; required in_process, CPU only.
  • Node READMEs follow docs/node_readme_template.md: yes.
  • Catalog regenerated: yes; all 211 existing rows unchanged, four new rows.
  • Discovery task list: added full_duplex; no bundle added.
  • Documentation: root EN/ZH tables, task index/guide, task/node READMEs, fixture
    rationale, and generated atlas updated. Local plans, translations, reports,
    experiments, raw data, and build outputs are not part of this PR.

Validation

On Windows / Python 3.11, with COLUMNS=160 and source imports from this branch:

  • Five acceptance files: 303 passed, no skips/failures.
  • Wider portable run: 874 passed, 67 skipped, 13 failed, with three previously
    confirmed Windows-path cases explicitly deselected. All 13 additional failures
    reproduce on clean base 204ffe8; 19 related source/test blobs are unchanged.
    They concern POSIX fake executables/bash/paths, Windows mmap file locks, and
    unavailable fcntl. This is not a claim that the full Windows suite passes.
  • CI-selected environment metadata tests: 16 passed, 16 deselected.
  • Full Ruff and whitespace checks passed; catalog/atlas regeneration verified.
  • Eight exact-route saved-stream replays / 40 CLI commands passed. All metrics,
    raw arrays and ledger audits match the earlier immutable reports.
  • Isolated wheel/sdist build and wheel asset checker passed; all 18 new production
    resources match source bytes in both artifacts. An actual local-wheel install
    outside the editable checkout passed CLI discovery/description/environment/
    scoring, including the exact IRD identity and 1700 ms / 4-event fixture result.
  • Linux and Python 3.10/3.12 validation is left to repository CI.

Commands used (evidence/build outputs were stored outside the checkout):

python -m pytest -q --ignore=tests/test_evaluation_env_check.py \
  -k 'not test_evaluate_predictions_remaps_workspace_audio_paths and not test_tts_script_run_injects_shared_transcriber_for_semantic_metric and not test_tts_script_run_selects_qwen3_asr_transcriber_by_pipeline_id'
python -m pytest tests/test_evaluation_env_check.py -q \
  -k 'in_process or pip_runtime or dry_run or doctor or cache_dir or se_pip_runtime_nodes_are_declared'
python -m ruff check .
git diff --check
python scripts/generate_pipeline_catalog.py
python scripts/generate_pipeline_atlas.py
uv build --python <prepared-python> --no-python-downloads --out-dir <external-dist>
python scripts/check_wheel_node_assets.py <built-wheel>

Exact route smoke:

sure-eval metric routes full_duplex --metric ird --json
sure-eval metric describe full_duplex \
  --pipeline-id full_duplex.any.ird.duplex_timeline_v1.fdbench_timing_v1 \
  --output <external-output>/pipeline.json
sure-eval env setup --pipeline <external-output>/pipeline.json --dry-run --json
sure-eval env check --pipeline <external-output>/pipeline.json --json
sure-eval metric run --pipeline <external-output>/pipeline.json \
  --reference-jsonl tests/fixtures/full_duplex/reference.jsonl \
  --sample-output tests/fixtures/full_duplex/sample_output.jsonl \
  --output-dir <external-output>/run --validate-env --json

Both report and pipeline description preserve the selected atomic identity.
Heavy-node inference smoke is not applicable: neither new node downloads or
loads a model. Source/fixture references identify the restricted upstream
project without vendoring its source or data or claiming its license is MIT.

Safety

  • No environments, checkpoints, generated run reports, caches, or credentials.
  • No private absolute paths in committed config, docs, tests, or fixtures.
  • Score-affecting rules declared in node/route identity and fixed protocol.
  • Existing defaults preserved; new-task discovery/defaults tested.

Copilot AI balanced review requested due to automatic review settings November 4, 2026 10:24

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has insufficient remaining quota. Try requesting a review at a lower effort level.

@PigeonDan1

Copy link
Copy Markdown
Collaborator

The original contributor fork is no longer available, so GitHub could not update this conflicting PR branch. The implementation was preserved with the original author attribution, rebased onto current main, and carried through maintainer PR #44. PR #44 passed all seven Core checks and has been merged. Closing this superseded PR.

@PigeonDan1 PigeonDan1 closed this Nov 12, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants