[aidless]

Closes #1483.

What

PassAtK.pass_at_k evaluated the unbiased estimator at self.n — the number of samples that was requested — while compute only passes the samples that actually exist (predictions[: self.n]). Every missing generation was therefore counted as a failure, so pass@k was a lower bound instead of an estimate whenever the backend returned fewer than n generations. compute only logged a warning for that case.

c: int = all_scores.count(1)
if self.n - c < self.k:          # self.n: requested
    return 1.0
return 1.0 - np.prod(1.0 - self.k / np.arange(self.n - c + 1, self.n + 1))

Fix

src/lighteval/metrics/metrics_sample.py, pass_at_k only:

n = len(all_scores)
if n == 0:
    return 0.0
if n - c < self.k:
    return 1.0
return 1.0 - np.prod(1.0 - self.k / np.arange(n - c + 1, n + 1))

• n >= samples is untouched: all_scores already has at most self.n entries, so len(all_scores) == self.n whenever enough samples came back.
• The n == 0 guard keeps "no samples at all" at 0.0 (with the naive rewrite it would flip to 1.0 because 0 - 0 < k).
• compute, the warning, and num_samples() are unchanged, so anything that reads the configured n still sees it.

If you disagree and consider "missing samples are failures" to be the intended contract, please say so on the issue and I'll drop this in favour of a docs/warning-only change — the numbers then remain a lower bound and should be documented as such.

Before / after

n=10 requested, 5 generations returned, 3 correct (c=3), measured through PassAtK.compute with a real Doc/ModelResponse:

k	before (b97a653)	after
1	0.30000000000000004	0.6
2	0.5333333333333332	0.9
3	0.7083333333333333	1.0

(0.6 / 0.9 / 1.0 are 1 - C(5-3, k) / C(5, k), i.e. the estimator over the five samples that exist.)

Tests

tests/test_unit_base_metrics.py:

• test_pass_at_k_with_fewer_samples_than_n — the table above.
• test_pass_at_k_with_n_samples_is_unchanged — same scores with n=5, plus a 10-sample case, both unchanged.
• test_pass_at_k_without_any_sample_is_zero — pass_at_k([]) == 0.0.

ruff check and ruff format --check pass on both files.

Local note: this checkout has no torch/transformers, so pytest tests/test_unit_base_metrics.py cannot be collected here (unrelated to this change — the module imports lighteval.models.model_output); the same three assertions were run through a standalone script that loads the untouched metrics_sample.py and prints the table above. CI will run the real file.

Out of scope

GPassAtK.g_pass_at_k (metrics_sample.py:1418) does n = self.n as well and has the same short-sample mismatch. It is not touched here to keep this PR to PassAtK; happy to follow up once this one settles. #1410 (k > n returning 1.0 for all-wrong samples) is a separate issue and is also left alone.