[Sivadheeraj5356]

Bug

Probability.compute in src/lighteval/metrics/metrics_sample.py never uses doc.gold_index: it aggregates np.exp(logprobs) over all choices, so with the default np.max aggregation it returns the probability of the best-scoring choice rather than the gold one. This contradicts the class docstring ("Returns the probability of choosing gold choice ignoring probabilities of other choices... aggregated probability of the gold choices"), ProbabilityMetric's docstring ("returns the probability of the gold choice given normalization"), and the sibling NormalizedMultiChoiceProbability, which indexes normalized_probs[gold_ixs].

Reproduction

doc = Doc(query="q", choices=["wrong", "right"], gold_index=1, task_name="t")
resp = ModelResponse(logprobs=np.log([0.9, 0.007]))
Probability().compute(doc, resp)  # 0.9 (best choice) - expected 0.007 (gold choice)

Fix

Filter to gold probabilities before aggregating, mirroring NormalizedMultiChoiceProbability (as_list handles both int and list gold indices):

probs = np.exp(normalized_log_probs)
gold_ixs = as_list(doc.gold_index)
return self.aggregation_function(probs[gold_ixs])

Tests

• Updated the existing aggregation case in test_prob, which encoded the current behavior by aggregating over non-gold choices; it now exercises aggregation over multiple gold indices.
• Added a regression case where the gold choice is not the argmax.
• Test run note: the full pytest suite was not run (heavy optional deps not installed locally). The fix and the new/updated cases were validated by importing the real, unmodified-except-this-fix metrics_sample.py with heavy third-party deps stubbed and asserting 4 numeric cases: single gold 0.7, multi-gold min-aggregation 0.1, gold-not-argmax 0.1 (regression), gold-is-argmax 0.7 unchanged.