[yunyv]

Describe the bug

Recall uses argmax at depth 1, which chooses the first tied choice, but reverses an ascending argsort for larger depths. A correct choice at depth 1 can therefore disappear at depth 2 even though the scores and gold label are unchanged.

To Reproduce

from lighteval.metrics.metrics_sample import Recall
from lighteval.models.model_output import ModelResponse
from lighteval.tasks.requests import Doc

doc = Doc(query="q", choices=["A", "B", "C"], gold_index=0)
response = ModelResponse(logprobs=[-1.0, -1.0, -1.0])
print([Recall(k).compute(doc, response) for k in (1, 2, 3)])

Before: [1, 0, 1]. After: [1, 1, 1].

Expected behavior

For valid floating-point log-probabilities (including negative infinity), use the same first-choice tie preference at every depth so top-k choices remain nested. This does not define NaN handling. Keep the existing argmax fast path at depth 1 and stably sort descending scores at larger depths. Gold-label handling, aggregation, API and dependencies are unchanged.

Regression tests cover all ties, ties at the cutoff, different gold positions, multiple golds, unequal scores, negative infinity, a single choice, zero depth and depth larger than the choice count. They also check monotonicity and equivalence with unnormalized log-probability accuracy at depth 1.

Version info

Reproduced through normal imports from an editable installation of main 7fa19dc6f53a713ba8bb6541e7ed66dad2e24fd7: LightEval 0.13.1.dev0, Python 3.12.14, NumPy 2.5.4, Linux.

Validation

• New regression tests on pristine production source: 6 failed, 7 passed.
• Patched regression tests: 13 passed.
• Combined base metrics, normalization, regression and selected offline automated-metric suites: 59 passed, 3 skipped, 14 deselected.
• Repository-wide make quality: passed.
• pre-commit run --all-files: passed.

The 59 passing cases come from tests/test_unit_base_metrics.py, tests/unit/metrics/test_normalizations.py, the new tests/unit/metrics/test_recall.py, and the selected JSON fixtures through tests/unit/metrics/test_automated_metrics_pytest.py. This is a selected offline run, not the full metrics suite.

The three existing skips are test_pass_at_k_estimator (existing “Need to understand what it does” placeholder), test_f1_score_quasi and test_f1 (existing NLTK placeholder tests).

The 14 deselections are:

• test_f1_dynamic_metric and test_exact_match_dynamic_metric: optional multilingual tokenization dependencies were not installed.
• Twelve JSON suites omitted to keep validation offline and avoid model/text-data downloads: bert_score, bleurt, extractiveness, faithfulness, bleu, bleu_1, bleu_4, rouge1, rouge2, rougeL, rougeLsum, and rouge_t5. No full GPU/dev-gpu CI or end-to-end benchmark was run. No separate type-check command is defined by the repository CI.

Prepared with AI assistance; no human review is claimed.