Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -203,7 +203,7 @@ packages and Git.
| TTS | CER, WER, speaker similarity, DNSMOS, WV-MOS, UTMOS, KOMOS (Korean) | [TTS](docs/tasks/tts.md) |
| TTA | FAD, FD, KL, IS, CLAP score, AudioBox Aesthetics CE/CU/PC/PQ | [TTA](docs/tasks/tta.md) |
| VC | CER, WER, speaker similarity, DNSMOS, WV-MOS, UTMOS | [VC](docs/tasks/vc.md) |
| SE | SI-SDR, STOI/eSTOI, PESQ, LSD, nSRMR, LPS, DNSMOS P.835/P.808 components, WV-MOS, UTMOS, NISQA DIM, SQUIM objective predictions | [SE](docs/tasks/se.md) |
| SE | SI-SDR, STOI/eSTOI, PESQ, LSD, nSRMR, LPS, DNSMOS P.835/P.808 components, WV-MOS, UTMOS, NISQA DIM, SQUIM objective/subjective MOS predictions, natural-speech SCOREQ, SQ-AST MOS and quality dimensions | [SE](docs/tasks/se.md) |
| RIR estimation | NPM, RT30/RT60 MAE, DRR/C50/C80/EDC, IACC full/early/late, ITD/ILD/IPD/DOA errors | [RIR estimation](docs/tasks/rir.md) |
| TSE | SI-SDR, speaker similarity, DNSMOS, WV-MOS, UTMOS, CER, WER | [TSE](docs/tasks/tse.md) |
| Classification / SER / GR | Accuracy | [Classification](docs/tasks/classification.md) |
Expand Down
2 changes: 1 addition & 1 deletion README_ZH.md
Original file line number Diff line number Diff line change
Expand Up @@ -192,7 +192,7 @@ node-local 虚拟环境只保留在本地,不会进入安装包或 Git。
| TTS | CER、WER、说话人相似度、DNSMOS、WV-MOS、UTMOS、KOMOS(韩语) | [TTS](docs/tasks/tts.md) |
| TTA | FAD、FD、KL、IS、CLAP score、AudioBox Aesthetics CE/CU/PC/PQ | [TTA](docs/tasks/tta.md) |
| VC | CER、WER、说话人相似度、DNSMOS、WV-MOS、UTMOS | [VC](docs/tasks/vc.md) |
| SE | SI-SDR、STOI/eSTOI、PESQ、LSD、nSRMR、LPS、DNSMOS P.835/P.808 分量、WV-MOS、UTMOS、NISQA DIM、SQUIM 预测指标 | [SE](docs/tasks/se.md) |
| SE | SI-SDR、STOI/eSTOI、PESQ、LSD、nSRMR、LPS、DNSMOS P.835/P.808 分量、WV-MOS、UTMOS、NISQA DIM、SQUIM 客观/主观 MOS 预测、自然语音 SCOREQ、SQ-AST MOS 和质量维度 | [SE](docs/tasks/se.md) |
| RIR estimation | NPM、RT30/RT60 MAE、DRR/C50/C80/EDC、IACC 全窗/早期/晚期、ITD/ILD/IPD/DOA 误差 | [RIR estimation](docs/tasks/rir.md) |
| TSE | SI-SDR、说话人相似度、DNSMOS、WV-MOS、UTMOS、CER、WER | [TSE](docs/tasks/tse.md) |
| Classification / SER / GR | Accuracy | [Classification](docs/tasks/classification.md) |
Expand Down
8 changes: 4 additions & 4 deletions docs/atlas/index.html

Large diffs are not rendered by default.

4 changes: 2 additions & 2 deletions docs/atlas/pipeline_atlas.svg
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
4 changes: 2 additions & 2 deletions docs/atlas/pipeline_atlas_dark.svg
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
7 changes: 7 additions & 0 deletions docs/pipeline_catalog.jsonl
Original file line number Diff line number Diff line change
Expand Up @@ -150,6 +150,13 @@
{"task": "SE", "task_alias": "se", "language": "n/a", "metric": "squim_stoi", "pipeline_id": "se.any.squim_stoi.squim_stoi_v1", "pipeline_kind": "atomic", "member_pipeline_ids": [], "execution_metrics": ["squim_stoi"], "nodes": ["scoring/squim_stoi"], "computation_node_ids": ["scoring/squim_stoi"], "task_config_path": "src/sure_eval/evaluation/tasks/se/manifest.yaml", "route_config_path": "src/sure_eval/evaluation/tasks/se/routes.yaml", "describe_entrypoint": "sure_eval.evaluation.scripts.se.describe_pipeline", "script_entrypoint": "sure_eval.evaluation.scripts.se.run", "executor": "sure_eval.evaluation.tasks.se.pipeline.evaluate_se_samples", "required_roles": ["enhanced_audio"], "optional_roles": ["noisy_audio", "reference_audio"]}
{"task": "SE", "task_alias": "se", "language": "n/a", "metric": "squim_pesq", "pipeline_id": "se.any.squim_pesq.squim_pesq_v1", "pipeline_kind": "atomic", "member_pipeline_ids": [], "execution_metrics": ["squim_pesq"], "nodes": ["scoring/squim_pesq"], "computation_node_ids": ["scoring/squim_pesq"], "task_config_path": "src/sure_eval/evaluation/tasks/se/manifest.yaml", "route_config_path": "src/sure_eval/evaluation/tasks/se/routes.yaml", "describe_entrypoint": "sure_eval.evaluation.scripts.se.describe_pipeline", "script_entrypoint": "sure_eval.evaluation.scripts.se.run", "executor": "sure_eval.evaluation.tasks.se.pipeline.evaluate_se_samples", "required_roles": ["enhanced_audio"], "optional_roles": ["noisy_audio", "reference_audio"]}
{"task": "SE", "task_alias": "se", "language": "n/a", "metric": "squim_si_sdr", "pipeline_id": "se.any.squim_si_sdr.squim_si_sdr_v1", "pipeline_kind": "atomic", "member_pipeline_ids": [], "execution_metrics": ["squim_si_sdr"], "nodes": ["scoring/squim_si_sdr"], "computation_node_ids": ["scoring/squim_si_sdr"], "task_config_path": "src/sure_eval/evaluation/tasks/se/manifest.yaml", "route_config_path": "src/sure_eval/evaluation/tasks/se/routes.yaml", "describe_entrypoint": "sure_eval.evaluation.scripts.se.describe_pipeline", "script_entrypoint": "sure_eval.evaluation.scripts.se.run", "executor": "sure_eval.evaluation.tasks.se.pipeline.evaluate_se_samples", "required_roles": ["enhanced_audio"], "optional_roles": ["noisy_audio", "reference_audio"]}
{"task": "SE", "task_alias": "se", "language": "n/a", "metric": "squim_mos", "pipeline_id": "se.any.squim_mos.squim_mos_v1", "pipeline_kind": "atomic", "member_pipeline_ids": [], "execution_metrics": ["squim_mos"], "nodes": ["scoring/squim_mos"], "computation_node_ids": ["scoring/squim_mos"], "task_config_path": "src/sure_eval/evaluation/tasks/se/manifest.yaml", "route_config_path": "src/sure_eval/evaluation/tasks/se/routes.yaml", "describe_entrypoint": "sure_eval.evaluation.scripts.se.describe_pipeline", "script_entrypoint": "sure_eval.evaluation.scripts.se.run", "executor": "sure_eval.evaluation.tasks.se.pipeline.evaluate_se_samples", "required_roles": ["enhanced_audio"], "optional_roles": ["noisy_audio", "reference_audio"]}
{"task": "SE", "task_alias": "se", "language": "n/a", "metric": "scoreq", "pipeline_id": "se.any.scoreq.scoreq_v1", "pipeline_kind": "atomic", "member_pipeline_ids": [], "execution_metrics": ["scoreq"], "nodes": ["scoring/scoreq"], "computation_node_ids": ["scoring/scoreq"], "task_config_path": "src/sure_eval/evaluation/tasks/se/manifest.yaml", "route_config_path": "src/sure_eval/evaluation/tasks/se/routes.yaml", "describe_entrypoint": "sure_eval.evaluation.scripts.se.describe_pipeline", "script_entrypoint": "sure_eval.evaluation.scripts.se.run", "executor": "sure_eval.evaluation.tasks.se.pipeline.evaluate_se_samples", "required_roles": ["enhanced_audio"], "optional_roles": ["noisy_audio", "reference_audio"]}
{"task": "SE", "task_alias": "se", "language": "n/a", "metric": "sq_ast_mos", "pipeline_id": "se.any.sq_ast_mos.sq_ast_mos_v1", "pipeline_kind": "atomic", "member_pipeline_ids": [], "execution_metrics": ["sq_ast_mos"], "nodes": ["scoring/sq_ast_mos"], "computation_node_ids": ["scoring/sq_ast_mos"], "task_config_path": "src/sure_eval/evaluation/tasks/se/manifest.yaml", "route_config_path": "src/sure_eval/evaluation/tasks/se/routes.yaml", "describe_entrypoint": "sure_eval.evaluation.scripts.se.describe_pipeline", "script_entrypoint": "sure_eval.evaluation.scripts.se.run", "executor": "sure_eval.evaluation.tasks.se.pipeline.evaluate_se_samples", "required_roles": ["enhanced_audio"], "optional_roles": ["noisy_audio", "reference_audio"]}
{"task": "SE", "task_alias": "se", "language": "n/a", "metric": "sq_ast_noi", "pipeline_id": "se.any.sq_ast_noi.sq_ast_noi_v1", "pipeline_kind": "atomic", "member_pipeline_ids": [], "execution_metrics": ["sq_ast_noi"], "nodes": ["scoring/sq_ast_noi"], "computation_node_ids": ["scoring/sq_ast_noi"], "task_config_path": "src/sure_eval/evaluation/tasks/se/manifest.yaml", "route_config_path": "src/sure_eval/evaluation/tasks/se/routes.yaml", "describe_entrypoint": "sure_eval.evaluation.scripts.se.describe_pipeline", "script_entrypoint": "sure_eval.evaluation.scripts.se.run", "executor": "sure_eval.evaluation.tasks.se.pipeline.evaluate_se_samples", "required_roles": ["enhanced_audio"], "optional_roles": ["noisy_audio", "reference_audio"]}
{"task": "SE", "task_alias": "se", "language": "n/a", "metric": "sq_ast_dis", "pipeline_id": "se.any.sq_ast_dis.sq_ast_dis_v1", "pipeline_kind": "atomic", "member_pipeline_ids": [], "execution_metrics": ["sq_ast_dis"], "nodes": ["scoring/sq_ast_dis"], "computation_node_ids": ["scoring/sq_ast_dis"], "task_config_path": "src/sure_eval/evaluation/tasks/se/manifest.yaml", "route_config_path": "src/sure_eval/evaluation/tasks/se/routes.yaml", "describe_entrypoint": "sure_eval.evaluation.scripts.se.describe_pipeline", "script_entrypoint": "sure_eval.evaluation.scripts.se.run", "executor": "sure_eval.evaluation.tasks.se.pipeline.evaluate_se_samples", "required_roles": ["enhanced_audio"], "optional_roles": ["noisy_audio", "reference_audio"]}
{"task": "SE", "task_alias": "se", "language": "n/a", "metric": "sq_ast_col", "pipeline_id": "se.any.sq_ast_col.sq_ast_col_v1", "pipeline_kind": "atomic", "member_pipeline_ids": [], "execution_metrics": ["sq_ast_col"], "nodes": ["scoring/sq_ast_col"], "computation_node_ids": ["scoring/sq_ast_col"], "task_config_path": "src/sure_eval/evaluation/tasks/se/manifest.yaml", "route_config_path": "src/sure_eval/evaluation/tasks/se/routes.yaml", "describe_entrypoint": "sure_eval.evaluation.scripts.se.describe_pipeline", "script_entrypoint": "sure_eval.evaluation.scripts.se.run", "executor": "sure_eval.evaluation.tasks.se.pipeline.evaluate_se_samples", "required_roles": ["enhanced_audio"], "optional_roles": ["noisy_audio", "reference_audio"]}
{"task": "SE", "task_alias": "se", "language": "n/a", "metric": "sq_ast_loud", "pipeline_id": "se.any.sq_ast_loud.sq_ast_loud_v1", "pipeline_kind": "atomic", "member_pipeline_ids": [], "execution_metrics": ["sq_ast_loud"], "nodes": ["scoring/sq_ast_loud"], "computation_node_ids": ["scoring/sq_ast_loud"], "task_config_path": "src/sure_eval/evaluation/tasks/se/manifest.yaml", "route_config_path": "src/sure_eval/evaluation/tasks/se/routes.yaml", "describe_entrypoint": "sure_eval.evaluation.scripts.se.describe_pipeline", "script_entrypoint": "sure_eval.evaluation.scripts.se.run", "executor": "sure_eval.evaluation.tasks.se.pipeline.evaluate_se_samples", "required_roles": ["enhanced_audio"], "optional_roles": ["noisy_audio", "reference_audio"]}
{"task": "SE", "task_alias": "se", "language": "n/a", "metric": "nisqa_mos", "pipeline_id": "se.any.nisqa_mos.nisqa_mos_v1", "pipeline_kind": "atomic", "member_pipeline_ids": [], "execution_metrics": ["nisqa_mos"], "nodes": ["scoring/nisqa_mos"], "computation_node_ids": ["scoring/nisqa_mos"], "task_config_path": "src/sure_eval/evaluation/tasks/se/manifest.yaml", "route_config_path": "src/sure_eval/evaluation/tasks/se/routes.yaml", "describe_entrypoint": "sure_eval.evaluation.scripts.se.describe_pipeline", "script_entrypoint": "sure_eval.evaluation.scripts.se.run", "executor": "sure_eval.evaluation.tasks.se.pipeline.evaluate_se_samples", "required_roles": ["enhanced_audio"], "optional_roles": ["noisy_audio", "reference_audio"]}
{"task": "SE", "task_alias": "se", "language": "n/a", "metric": "nisqa_noi", "pipeline_id": "se.any.nisqa_noi.nisqa_noi_v1", "pipeline_kind": "atomic", "member_pipeline_ids": [], "execution_metrics": ["nisqa_noi"], "nodes": ["scoring/nisqa_noi"], "computation_node_ids": ["scoring/nisqa_noi"], "task_config_path": "src/sure_eval/evaluation/tasks/se/manifest.yaml", "route_config_path": "src/sure_eval/evaluation/tasks/se/routes.yaml", "describe_entrypoint": "sure_eval.evaluation.scripts.se.describe_pipeline", "script_entrypoint": "sure_eval.evaluation.scripts.se.run", "executor": "sure_eval.evaluation.tasks.se.pipeline.evaluate_se_samples", "required_roles": ["enhanced_audio"], "optional_roles": ["noisy_audio", "reference_audio"]}
{"task": "SE", "task_alias": "se", "language": "n/a", "metric": "nisqa_dis", "pipeline_id": "se.any.nisqa_dis.nisqa_dis_v1", "pipeline_kind": "atomic", "member_pipeline_ids": [], "execution_metrics": ["nisqa_dis"], "nodes": ["scoring/nisqa_dis"], "computation_node_ids": ["scoring/nisqa_dis"], "task_config_path": "src/sure_eval/evaluation/tasks/se/manifest.yaml", "route_config_path": "src/sure_eval/evaluation/tasks/se/routes.yaml", "describe_entrypoint": "sure_eval.evaluation.scripts.se.describe_pipeline", "script_entrypoint": "sure_eval.evaluation.scripts.se.run", "executor": "sure_eval.evaluation.tasks.se.pipeline.evaluate_se_samples", "required_roles": ["enhanced_audio"], "optional_roles": ["noisy_audio", "reference_audio"]}
Expand Down
28 changes: 24 additions & 4 deletions docs/tasks/se.md
Original file line number Diff line number Diff line change
Expand Up @@ -26,6 +26,13 @@ sure-eval metric routes se --metric dnsmos
| `squim_stoi` | `squim_stoi` | `se.any.squim_stoi.squim_stoi_v1` | `scoring/squim_stoi` | `enhanced_audio` |
| `squim_pesq` | `squim_pesq` | `se.any.squim_pesq.squim_pesq_v1` | `scoring/squim_pesq` | `enhanced_audio` |
| `squim_si_sdr` | `squim_si_sdr` | `se.any.squim_si_sdr.squim_si_sdr_v1` | `scoring/squim_si_sdr` | `enhanced_audio` |
| `squim_mos` | `squim_mos` | `se.any.squim_mos.squim_mos_v1` | `scoring/squim_mos` | `enhanced_audio` |
| `scoreq` | `scoreq` | `se.any.scoreq.scoreq_v1` | `scoring/scoreq` | `enhanced_audio` |
| `sq_ast_mos` | `sq_ast_mos` | `se.any.sq_ast_mos.sq_ast_mos_v1` | `scoring/sq_ast_mos` | `enhanced_audio` |
| `sq_ast_noi` | `sq_ast_noi` | `se.any.sq_ast_noi.sq_ast_noi_v1` | `scoring/sq_ast_noi` | `enhanced_audio` |
| `sq_ast_dis` | `sq_ast_dis` | `se.any.sq_ast_dis.sq_ast_dis_v1` | `scoring/sq_ast_dis` | `enhanced_audio` |
| `sq_ast_col` | `sq_ast_col` | `se.any.sq_ast_col.sq_ast_col_v1` | `scoring/sq_ast_col` | `enhanced_audio` |
| `sq_ast_loud` | `sq_ast_loud` | `se.any.sq_ast_loud.sq_ast_loud_v1` | `scoring/sq_ast_loud` | `enhanced_audio` |
| `nisqa_mos` | `nisqa_mos` | `se.any.nisqa_mos.nisqa_mos_v1` | `scoring/nisqa_mos` | `enhanced_audio` |
| `nisqa_noi` | `nisqa_noi` | `se.any.nisqa_noi.nisqa_noi_v1` | `scoring/nisqa_noi` | `enhanced_audio` |
| `nisqa_dis` | `nisqa_dis` | `se.any.nisqa_dis.nisqa_dis_v1` | `scoring/nisqa_dis` | `enhanced_audio` |
Expand All @@ -51,10 +58,20 @@ Existing `dnsmos` remains the calibrated P.835-style **OVRL** score. New
`dnsmos_sig`, `dnsmos_bak`, and `dnsmos_p808` select the provider's calibrated
SIG, BAK, and P808_MOS outputs; they do not reroute or change old `dnsmos`
scores. These are learned predictions, not P.835/P.808 human listening tests.
`squim_*` are TorchAudio SQUIM **no-reference predictions** of the named
reference metrics; they cannot be mixed with `stoi`, `pesq`, or `si_sdr` as if
they used the same measurement definition. `nisqa_*` are the five DIM outputs
`squim_stoi`, `squim_pesq`, and `squim_si_sdr` are TorchAudio SQUIM
**no-reference predictions** of the named reference metrics; they cannot be
mixed with `stoi`, `pesq`, or `si_sdr` as if they used the same measurement
definition. `squim_mos` is TorchAudio SQUIM Subjective: it predicts MOS from
the enhanced waveform and a separately staged clean non-matching reference,
so it is not a human listening test or a pure no-reference model. `scoreq` is
the natural-speech no-reference SCOREQ MOS predictor. `nisqa_*` are the five DIM outputs
from the same NISQA v2 checkpoint, with one corpus mean per output.
`sq_ast_*` are SQ-AST's five non-intrusive predictions using the improved
weights: overall quality, noisiness, discontinuity, coloration, and loudness.
Each is mapped from a sigmoid output as `4 * output + 1` to a 1–5
ACR-like scale. Audio is downmixed and resampled to 48 kHz; the official
feature-extractor normalization is used. The published operating range is
4–12 seconds per clip, and scores outside it should be treated as exploratory.

POLQA/P.863 is **not** registered: its white paper is not a licensed scoring
implementation. Do not report a proxy as POLQA.
Expand Down Expand Up @@ -119,7 +136,10 @@ lightweight nodes require `pystoi` (`estoi`), `librosa` (`lsd`), or
| Family | Runtime asset | Environment variable | Material restriction |
| --- | --- | --- | --- |
| `lps` | Pinned `facebook/wav2vec2-lv-60-espeak-cv-ft` directory | `SURE_SE_LPS_MODEL_DIR` | `transformers==4.24.0` legacy weight-norm compatibility |
| `squim_*` | TorchAudio `squim_objective_dns2020.pth` | `SURE_SE_SQUIM_OBJECTIVE_CKPT` | Objective weights CC BY 4.0 |
| `squim_stoi`, `squim_pesq`, `squim_si_sdr` | TorchAudio `squim_objective_dns2020.pth` | `SURE_SE_SQUIM_OBJECTIVE_CKPT` | Objective weights CC BY 4.0 |
| `squim_mos` | TorchAudio `squim_subjective_bvcc_daps.pth` plus a clean non-matching WAV | `SURE_SE_SQUIM_SUBJECTIVE_CKPT`, `SURE_SE_SQUIM_SUBJECTIVE_NMR` | Subjective weights CC BY-NC 4.0 |
| `scoreq` | Natural-speech `adapt_nr_telephone.onnx` | `sure-eval env download --node scoring/scoreq` or `SURE_SE_SCOREQ_CHECKPOINT` | Verify model asset terms |
| `sq_ast_*` | Improved `mos/noi/dis/col/loud.pth` heads and pinned `MIT/ast-finetuned-audioset-10-10-0.4593` snapshot | `SURE_SE_SQ_AST_*_CKPT`, `SURE_SE_SQ_AST_BASE_MODEL_DIR` | Repository code MIT; base model BSD-3-Clause; improved head asset terms unspecified |
| `nisqa_*` | NISQA DIM `nisqa.tar` | `SURE_SE_NISQA_DIM_CKPT` | Model weights CC BY-NC-SA 4.0; verify permitted use |

No checkpoint, local virtual environment, or cached result belongs in the PR.
Expand Down
37 changes: 37 additions & 0 deletions scripts/run_se_metric_pipeline.py
Original file line number Diff line number Diff line change
Expand Up @@ -56,6 +56,13 @@ def _build_stub_providers() -> tuple[dict[str, Any], dict[str, Any]]:
"squim_stoi": lambda prediction, reference="", **kwargs: {"squim_stoi": 0.89},
"squim_pesq": lambda prediction, reference="", **kwargs: {"squim_pesq": 2.9},
"squim_si_sdr": lambda prediction, reference="", **kwargs: {"squim_si_sdr": 10.0},
"squim_mos": lambda prediction, reference="", **kwargs: {"squim_mos": 3.7},
"scoreq": lambda prediction, reference="", **kwargs: {"scoreq": 3.8},
"sq_ast_mos": lambda prediction, reference="", **kwargs: {"sq_ast_mos": 3.9},
"sq_ast_noi": lambda prediction, reference="", **kwargs: {"sq_ast_noi": 4.0},
"sq_ast_dis": lambda prediction, reference="", **kwargs: {"sq_ast_dis": 3.7},
"sq_ast_col": lambda prediction, reference="", **kwargs: {"sq_ast_col": 4.1},
"sq_ast_loud": lambda prediction, reference="", **kwargs: {"sq_ast_loud": 4.2},
"nisqa_mos": lambda prediction, reference="", **kwargs: {"nisqa_mos": 3.8},
"nisqa_noi": lambda prediction, reference="", **kwargs: {"nisqa_noi": 3.6},
"nisqa_dis": lambda prediction, reference="", **kwargs: {"nisqa_dis": 3.7},
Expand Down Expand Up @@ -106,6 +113,24 @@ def _build_real_providers(*, device: str, cache_dir: Path, metrics: tuple[str, .
squim_provider = SQUIMObjectiveProvider(device=device)
for metric in squim_metrics:
mos_providers[metric] = squim_provider
if "squim_mos" in metrics:
from sure_eval.evaluation.nodes.scoring._squim_subjective import SQUIMSubjectiveProvider

mos_providers["squim_mos"] = SQUIMSubjectiveProvider(device=device)
if "scoreq" in metrics:
from sure_eval.evaluation.nodes.scoring._scoreq import SCOREQProvider

mos_providers["scoreq"] = SCOREQProvider(device=device)
sq_ast_metrics = set(metrics) & {"sq_ast_mos", "sq_ast_noi", "sq_ast_dis", "sq_ast_col", "sq_ast_loud"}
if sq_ast_metrics:
from sure_eval.evaluation.nodes.scoring._sq_ast import SQASTProvider

sq_ast_provider = SQASTProvider(
dimensions=tuple(sorted(metric.removeprefix("sq_ast_") for metric in sq_ast_metrics)),
device=device,
)
for metric in sq_ast_metrics:
mos_providers[metric] = sq_ast_provider
nisqa_metrics = set(metrics) & {"nisqa_mos", "nisqa_noi", "nisqa_dis", "nisqa_col", "nisqa_loud"}
if nisqa_metrics:
from sure_eval.evaluation.nodes.scoring._nisqa_dim import NISQADIMProvider
Expand Down Expand Up @@ -162,6 +187,18 @@ def _normalize_metric(metric: str) -> str:
"squim-stoi": "squim_stoi",
"squim-pesq": "squim_pesq",
"squim-si-sdr": "squim_si_sdr",
"squim-mos": "squim_mos",
"scoreq": "scoreq",
"sqast-mos": "sq_ast_mos",
"sqast-noi": "sq_ast_noi",
"sqast-dis": "sq_ast_dis",
"sqast-col": "sq_ast_col",
"sqast-loud": "sq_ast_loud",
"sq-ast-mos": "sq_ast_mos",
"sq-ast-noi": "sq_ast_noi",
"sq-ast-dis": "sq_ast_dis",
"sq-ast-col": "sq_ast_col",
"sq-ast-loud": "sq_ast_loud",
"nisqa-mos": "nisqa_mos",
"nisqa-noi": "nisqa_noi",
"nisqa-dis": "nisqa_dis",
Expand Down
Loading
Loading