diff --git a/README.md b/README.md index 6fb9d05..bc5b427 100644 --- a/README.md +++ b/README.md @@ -135,18 +135,92 @@ deployment parameters, and capability boundary beside the implementation: ## Reference results -These reference runs cover 2,081 cases. Operation success averages requested -operations; all-operations success requires every requested operation in a case -to succeed. AuK uses the official Prompt Enhancer. - -| Candidate | One-take operation success | Sequential operation success | One-take all operations | Sequential all operations | -|---|---:|---:|---:|---:| -| dots.tts.edit | **63.9%** | **62.7%** | **48.8%** | **48.2%** | -| MiMo-Audio-Instruct | 37.3% | 39.1% | 25.6% | 26.8% | -| Step-Audio-EditX | 36.0% | 31.1% | 24.9% | 20.6% | -| Ming-UniAudio | 18.7% | 18.8% | 8.7% | 10.5% | -| FireRedTTS3-Instruct | 16.0% | 20.5% | 8.3% | 11.6% | -| AuK | 23.6% | 33.3% | 12.3% | 21.9% | +Precise speech editing requires both successful execution of the requested +operations and preservation of the source speech. These reference results cover +all 2,081 cases across the five editing categories. Each value is the arithmetic +mean of the one-take and sequential results. Bold indicates the best result among +the listed systems. AuK uses the official Prompt Enhancer. + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
CandidateInstruction FollowingLocal Preservation
Operation success (%) ↑All success (%) ↑SIM ↑WDTW-Dur (%) ↓WDTW-F0 (st) ↓
dots.tts.edit63.3048.490.7797.181.84
MiMo-Audio-Instruct38.2026.210.62315.254.63
Step-Audio-EditX33.5722.730.71113.663.41
AuK28.4317.110.7599.402.92
Ming-UniAudio18.739.610.51413.194.19
FireRedTTS3-Instruct18.269.970.7777.762.01
+ +Operation success measures the fraction of requested operations that succeed; +All success measures the fraction of cases in which every requested operation +succeeds. SIM measures speaker similarity between source and edited speech. +WDTW-Dur and WDTW-F0 measure duration and pitch changes over regions designated +for preservation by the instruction; st denotes semitones. + +Within each execution mode, operation success is pooled over requested operations, +while All success and SIM weight cases equally. WDTW metrics pool distances and +normalization terms before division. The two execution-mode scores are then averaged. + +On the 240 compositional editing cases, sequential execution increases +dots.tts.edit's operation success from 68.12% to 68.48% and All success from 37.92% +to 42.92%. One-take execution achieves higher speaker similarity and lower WDTW errors. The radars show instruction following, preservation, and quality. Compositional axes report Component and All Success ([evaluation protocol](docs/evaluation.md)).