Skip to content

Add a Count column with the evaluated sample count to the markdown results summary - #1480

Open
amkkoussay wants to merge 1 commit into
huggingface:mainfrom
amkkoussay:804-count-column
Open

amkkoussay wants to merge 1 commit into
huggingface:mainfrom
amkkoussay:804-count-column

Conversation

@amkkoussay

Copy link
Copy Markdown

Fixes #804

What this does

The markdown results summary (make_results_table) now shows a Count column with the number of samples that were actually evaluated for each task, exported by the evaluation tracker as num_samples alongside results.

Design choices

  • Source of the count: the samples actually logged per task (DetailsLogger.details) — i.e. the filtered/evaluated set behind the metrics, not the raw dataset size. This follows the concern raised on the issue thread about counts being misleading when filtering is applied.
  • Placement: the count appears once per task, on its first metric row; the remaining metric rows stay blank so the table stays readable.
  • Backward compatibility: results dicts without counts (e.g. loaded from older runs) render a blank cell — no key is required and nothing breaks.

Changes

  • src/lighteval/logging/evaluation_tracker.py — export num_samples in both results-dict builders (results property and generate_final_dict).
  • src/lighteval/utils/utils.py — render the Count column in make_results_table.
  • tests/unit/utils/test_utils.py — new unit tests for the column.

Tests

  • Reproduction of the issue's example: before the change the table had no count; after the change the Count column shows the evaluated samples.
  • tests/unit/utils/test_utils.py: 9/9 pass (6 existing + 3 new covering: count on the task's first metric row only, missing counts, partial counts).

Happy to adjust the key name, column position, or the source of the count if you prefer a different design.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[FT] showing count in Markdown summary table

1 participant