Repository navigation
tests: fix harness metric comparison under pytest 9.2 (nested pytest.approx) - #1456
Merged
Merged
Conversation
pytest 9.2.0 raises TypeError when pytest.approx() is given nested data structures. Corpus-level metric inputs (mcc, loglikelihood_f1, multi_f1_numeric, target_perplexity, bits_per_byte) are compared as lists of dicts containing lists, so test_unit_harness_metrics failed for them (CI stopped on the first one, mcc, because of `pytest -x`). Add a small recursive approx_equal helper that walks dicts/lists and only applies pytest.approx to numeric leaves, with plain equality otherwise. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update. |
Cai-z-us
added a commit
to rwkv-rs/lighteval-rwkv
that referenced
this pull request
Nov 2, 2026
* fix: exclude compromised LiteLLM 1.107.0 (huggingface#1445) * fix: exclude compromised LiteLLM 1.107.0 * fix: use supported Hub model search argument * tests: make harness metric comparison work with pytest 9.2 (huggingface#1456) pytest 9.2.0 raises TypeError when pytest.approx() is given nested data structures. Corpus-level metric inputs (mcc, loglikelihood_f1, multi_f1_numeric, target_perplexity, bits_per_byte) are compared as lists of dicts containing lists, so test_unit_harness_metrics failed for them (CI stopped on the first one, mcc, because of `pytest -x`). Add a small recursive approx_equal helper that walks dicts/lists and only applies pytest.approx to numeric leaves, with plain equality otherwise. Co-authored-by: Nathan Habib <nathan.habib@hf.co> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> * Bump nltk from 3.10.3 to 3.10.4 (huggingface#1455) Co-authored-by: katsugtgz <katsugtgz@users.noreply.github.com> Co-authored-by: Nathan Habib <30601243+NathanHB@users.noreply.github.com> * refactor(rwkv): isolate native evaluation policy and publication * docs: keep custom task guide unchanged in PR * fix(rwkv): restore file-based score loading and a single average metric * fix(rwkv): restore direct score publication and standalone reader * refactor(rwkv): simplify model adapter overrides * fix(rwkv): bound retries and hand off terminal failures --------- Co-authored-by: Guillaume Salou <17745322+jagwar@users.noreply.github.com> Co-authored-by: Nathan Habib <30601243+NathanHB@users.noreply.github.com> Co-authored-by: Nathan Habib <nathan.habib@hf.co> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: Muhammad Naufal Mahdi <053792227@ecampus.ut.ac.id> Co-authored-by: katsugtgz <katsugtgz@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
pytest 9.2.0 (released Oct 27) raises
TypeError: pytest.approx() does not support nested data structures.tests/test_unit_harness_metrics.pycompared corpus-level metric inputs this way: lists of dicts containing lists, for mcc, loglikelihood_f1, multi_f1_numeric, target_perplexity and bits_per_byte.This has been breaking the Tests workflow on main since Oct 28, and it also blocks #1455 (nltk security bump). CI only reported mcc because it runs with
-x; 32 harness cases were failing.This adds a small recursive
approx_equalhelper. It walks dicts and lists and appliespytest.approxonly to numeric leaves, with plain equality for everything else. Metric code and reference scores are unchanged.Verified locally: 250/250 harness tests pass under both pytest 9.1.1 and 9.2.0; ruff check + format pass.
🤖 Generated with Claude Code