Repository navigation
Combine subtask stderrs in quadrature for the _average and all rows - #1468
Open
Andreas-Hum wants to merge 1 commit into
Open
Andreas-Hum wants to merge 1 commit into
Andreas-Hum wants to merge 1 commit into
Conversation
The stderr of the _average and all rows was the arithmetic mean of the subtask stderrs. These rows are unweighted means of G subtask scores computed on disjoint samples, so the standard error of the average is sqrt(sum_i stderr_i**2) / G. For the 57 MMLU subjects the mean of the stderrs is about 7x too large. Point estimates are unchanged. Fixes huggingface#1466
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #1466
What
The
_averagerows (e.g.mmlu:_average|5) and theallrow report the arithmetic mean of the subtask stderrs as the stderr of the averaged score. These rows are unweighted means of G subtask scores computed on disjoint question sets, so the standard error of the average issqrt(sum_i se_i**2) / G. lm-evaluation-harness'sunweighted_mean_stderruses the same formula. With the 57 MMLU subjects, the current value is about 7x too large: 0.0345 instead of 0.0047 in the example in the issue, which matches a Monte Carlo over fresh question draws.This PR changes
MetricsLogger.aggregateto add the_stderrvalues in quadrature for the_averagerows and forall. Point estimates are unchanged.For
all, this assumes the tasks in a run are evaluated on different samples. That holds unless the same benchmark appears twice, e.g. at two few-shot settings.Tests
tests/unit/logging/test_info_loggers.pychecks both rows againstsqrt(sum_i se_i**2) / G. It fails on currentmainand passes with this change.pytest tests/unit/logging: 9 passed, 1 skipped.ruff checkandruff format --check(ruff 0.11.10, as in.pre-commit-config.yaml) pass on the changed files.