Repository navigation
Conversation
CorpusLevelF1Score computed one binary F1 per class and averaged those, which is not the multiclass micro/macro/weighted F1 the caller asked for. For golds [1, 2, 0] and preds [0, 0, 0] the old code returned 0.5556, while the multiclass micro F1 is 0.3333. The multi_f1_numeric (mf1) test fixture encoded the old values, so its poor-performance and random-performance expectations are updated to the correct 0.0 and 1/3. The average=None path and the num_classes == 2 branch are untouched.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this changes (and what numbers move)
Before this change,
CorpusLevelF1Scorewithnum_classes > 2computed one binary F1 per class with a one-vs-rest probe and averaged those numbers. That is not multiclass micro/macro/weighted F1: a one-vs-rest probe for a class that is never predicted scores 1.0 when nothing at all is predicted, which inflates the average.After this change the requested averaging is delegated to
sklearn.metrics.f1_score(..., average=..., labels=range(num_classes)).Three-class corpus, golds -> preds:
[2,2,2] -> [2,2,2][1,0,0] -> [1,0,0][0,1,1] -> [1,0,0][1,2,0] -> [0,0,0]Existing results that change. Every
mf1score produced byMetrics.multi_f1_numeric(used bysuper_glue:cb,src/lighteval/tasks/tasks/glue.py:350) is affected, including scores already published in model cards, leaderboards or reports. The old numbers are inflated, worst when one class is never predicted.f1_score,f1_score_macro,f1_score_microandloglikelihood_f1are unchanged: thenum_classes == 2branch and theaverage=Nonepath are untouched.The
multi_f1_numericfixture encoded the old behaviour, so its two wrong expectations are corrected:0.33 -> 0.0(no true positives) and0.55 -> 1/3(one of three golds matched). The.gitattributesentry markstests/unit/metrics/test_cases/*.jsonas-diff, so the fixture change does not render as a text diff here; those two lines are the only content change.Tests
New
tests/unit/metrics/test_corpus_f1.py(7 cases): multiclass corpus compared againstsklearnformicro,macroandweighted; micro vs macro differ on the same corpus; all-zero predictions; single-class predictions; class absent from the corpus.Local results:
tests/unit/metrics/test_corpus_f1.py: 7 passedtests/unit/metrics/test_automated_metrics_pytest.py(incl.multi_f1_numeric): 45 passed, 1 failed (extractiveness, missinglighteval[multilingual]extra locally), 1 deselected (bleurt, hangs downloading its checkpoint)tests/unit/metrics/test_normalizations.py: 4 passedtests/unit/metrics/test_extractive_match.py: 233 passedThe
extractivenessfailure, thebleurthang and thetest_metric_requests.pyhang intest_pmi_requestall reproduce on the unmodified baseline in this environment and do not touch F1.