Skip to content

feat(se): add SQ-AST and non-intrusive quality metrics on latest main - #42

Open
Yj-Xiong wants to merge 1 commit into
X-LANCE:mainfrom
Yj-Xiong:feat/sq-ast-latest
Open

Yj-Xiong wants to merge 1 commit into
X-LANCE:mainfrom
Yj-Xiong:feat/sq-ast-latest

Conversation

@Yj-Xiong

Copy link
Copy Markdown

Includes five SQ-AST dimensions, SQUIM subjective MOS, SCOREQ, and refreshed pipeline declarations.

Start Here

Read Contributing and
Add Evaluation Capabilities.

Summary

Add seven opt-in SE metrics:

  • squim_mos: TorchAudio SQUIM Subjective predicted MOS using a separately staged clean non-matching reference.
  • scoreq: natural-speech no-reference SCOREQ predicted MOS.
  • sq_ast_mos, sq_ast_noi, sq_ast_dis, sq_ast_col, sq_ast_loud: SQ-AST overall quality and four perceptual dimensions using the improved weights.

All metrics preserve per-sample scores and arithmetic-mean aggregation. Existing SE default routes are unchanged.

PR Type

Guide followed: docs/pr_guides/new_metric.md and docs/contributing.md.

Evaluation Identity

  • Reported metric:
    • squim_mos
    • scoreq
    • sq_ast_mos
    • sq_ast_noi
    • sq_ast_dis
    • sq_ast_col
    • sq_ast_loud
  • execution_metrics, if different: same as reported metrics.
  • Added pipeline_id:
    • se.any.squim_mos.squim_mos_v1
    • se.any.scoreq.scoreq_v1
    • se.any.sq_ast_mos.sq_ast_mos_v1
    • se.any.sq_ast_noi.sq_ast_noi_v1
    • se.any.sq_ast_dis.sq_ast_dis_v1
    • se.any.sq_ast_col.sq_ast_col_v1
    • se.any.sq_ast_loud.sq_ast_loud_v1
  • Default route changed: no
  • Score comparability changed: no

Changed Declarations

  • Task manifest: yes
  • Task routes: yes
  • Node manifest: yes
  • node_env.yaml: yes
  • Node README follows docs/node_readme_template.md: yes
  • Catalog regenerated: yes
  • Catalog discovery task or curated bundle changed: no
  • Documentation coverage checked against docs/contributing.md: yes. Root EN/ZH READMEs, SE task documentation, all new node READMEs, generated catalog, and generated atlas were updated. The task index was unchanged because SE already exists.

The branch also includes the latest upstream main changes from ETC canonicalization. Final branch: Yj-Xiong:feat/sq-ast-latest.

Validation

Exact commands:

git fetch origin main --prune
git rebase origin/main

PYTHONNOUSERSITE=1 PYTHONPATH=src \
python -m pytest -q \
  tests/test_se_openbench_metrics.py \
  tests/test_se_pipeline_nodes.py \
  tests/test_pipeline_catalog_identity.py \
  tests/test_etc_canonicalization_node.py

PYTHONPATH=src sure-eval metric routes se \
  --metric sq_ast_mos --json

PYTHONPATH=src sure-eval metric describe se \
  --pipeline-id se.any.sq_ast_mos.sq_ast_mos_v1 \
  --output /tmp/sq_ast_mos.json --json

PYTHONPATH=src sure-eval env setup \
  --pipeline /tmp/sq_ast_mos.json --dry-run --json

PYTHONPATH=src python scripts/generate_pipeline_catalog.py
PYTHONPATH=src python scripts/generate_pipeline_atlas.py

ruff check .
git diff --check

The same route, describe, and environment checks passed for all five SQ-AST dimensions.

Validation result:

40 passed, 4 skipped

The skipped tests are optional heavy/model checks.

Real model smoke tests

  • SCOREQ:
    • Official checkpoint MD5: 00f512e40db4e447df54d73150efbebe
    • SURE score: 3.5275135040283203
    • Direct ONNX score: 3.5275135040283203
  • SQ-AST:
    • All five improved checkpoint hashes verified.
    • Official two-sample conformance passed for MOS, NOI, DIS, COL, and LOUD.
    • All dimensions were within 0.05 of the official README values.
    • SURE bundle report included all five metrics and per-sample scores.
    • Exact CLI describe -> run -> report for se.any.sq_ast_mos.sq_ast_mos_v1 passed.
  • SQUIM Subjective:
    • Route and provider contract tests passed.
    • Real checkpoint smoke was skipped because a separately staged non-matching reference WAV was unavailable.

The wheel build/install smoke also passed and contained all five SQ-AST nodes, manifests, environment declarations, READMEs, and routes.

Safety

  • No .venv, checkpoints, generated reports, local cache paths, or credentials committed
  • No private absolute paths in committed config or docs
  • Score-affecting behavior is declared in route/node identity
  • Existing default route behavior is tested and unchanged

Includes five SQ-AST dimensions, SQUIM subjective MOS, SCOREQ, and refreshed pipeline declarations.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants