Repository navigation
feat(full_duplex): add FD-Bench timing evaluation - #41
Closed
wangxiangruirui wants to merge 1 commit into
Closed
wangxiangruirui wants to merge 1 commit into
wangxiangruirui wants to merge 1 commit into
Conversation
Collaborator
|
The original contributor fork is no longer available, so GitHub could not update this conflicting PR branch. The implementation was preserved with the original author attribution, rebased onto current main, and carried through maintainer PR #44. PR #44 passed all seven Core checks and has been merged. Closing this superseded PR. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Add a
full_duplextask for scoring saved, shared-clock user/system speechtimelines with IRD, FSED, ERT, and EIT. Two reusable, CPU-only in-process nodes
validate the JSONL contract and calculate the four metrics in one traversal;
each atomic route selects one primary score while retaining all four results.
The independent implementation targets the fixed FD-Bench procedure at commit
8a4b7df1b4dcb0fc50a7a5660247a2ffe8d394eb. It preserves strict boundaries,candidate overwrites, special IRD endpoints, positive-tick filtering,
per-event integer-ms truncation, and pooled medians. Empty pools retain the
compatibility
0.0value with explicitno_eventsand measurement warnings.Reports retain detailed arrays and the event ledger once, with compact node
traces. Optional validated measurement provenance records VAD/playback/reference
settings without changing scores; missing/partial provenance remains valid v1
input and produces a comparability warning. JSON and terminal summaries
distinguish empty pools from observed sub-millisecond events reported as 0 ms.
No model serving, VAD inference, API integration, upstream source, benchmark
recordings, weights, or new runtime dependencies are included. The fixtures are
synthetic, including 34 frozen cases generated from the hash-locked official
timing oracle; normal CI reads only their inputs and expected outputs.
PR Type
Guides followed: new task, new metric, node change, and contributing.
Evaluation Identity
ird,fsed,ert,eit.full_duplex.any.ird.duplex_timeline_v1.fdbench_timing_v1full_duplex.any.fsed.duplex_timeline_v1.fdbench_timing_v1full_duplex.any.ert.duplex_timeline_v1.fdbench_timing_v1full_duplex.any.eit.duplex_timeline_v1.fdbench_timing_v1normalization/duplex_timeline@v1,scoring/fdbench_timing@v1.to IRD. FSED-filtered discovery retains the task's IRD default metadata.
direction. Empty-pool zero is not a measured zero latency. Playback and VAD
provenance must be controlled before comparing saved recordings.
Changed Declarations
node_env.yaml: added; requiredin_process, CPU only.docs/node_readme_template.md: yes.full_duplex; no bundle added.rationale, and generated atlas updated. Local plans, translations, reports,
experiments, raw data, and build outputs are not part of this PR.
Validation
On Windows / Python 3.11, with
COLUMNS=160and source imports from this branch:confirmed Windows-path cases explicitly deselected. All 13 additional failures
reproduce on clean base
204ffe8; 19 related source/test blobs are unchanged.They concern POSIX fake executables/bash/paths, Windows mmap file locks, and
unavailable
fcntl. This is not a claim that the full Windows suite passes.raw arrays and ledger audits match the earlier immutable reports.
resources match source bytes in both artifacts. An actual local-wheel install
outside the editable checkout passed CLI discovery/description/environment/
scoring, including the exact IRD identity and 1700 ms / 4-event fixture result.
Commands used (evidence/build outputs were stored outside the checkout):
Exact route smoke:
Both report and pipeline description preserve the selected atomic identity.
Heavy-node inference smoke is not applicable: neither new node downloads or
loads a model. Source/fixture references identify the restricted upstream
project without vendoring its source or data or claiming its license is MIT.
Safety