Repository navigation
Conversation
calib_err compared confidences in percent with 0/1 correctness. Divide by 100 before the call and scale the result back, as the official HLE judge does.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #1469
What was wrong
JudgeLLMHLE.compute_corpuspassed the judge's confidence (0 to 100) and correctness (0 or 1) tocalib_errwithout bringing them to the same scale. The reportedcalibration_errortherefore mostly measured how confident the model sounds. A model that is always correct at 100% confidence scored 70.0 instead of 0.Fix
Divide the confidences by 100 before
calib_errand multiply the result by 100, as the official HLE judge script does. The binning is unchanged.How it was verified
In a Linux container (CPU only), numpy 2.3.4, on main
311c9d04f4173791ff97bf2d3bf811ec057aa799:tests/unit/tasks/test_hle.py: 200 correct answers at 100% confidence givecalibration_error == 0.0. On main it fails with70.0; with the fix it passes (pytest tests/unit/tasks/test_hle.py, lighteval installed withpip install -e .).The rest of the test suite and the lint hooks were not run.