[jhwanseok]

Adds HAE-RAE Bench, the Korean knowledge benchmark (https://arxiv.org/abs/2309.02706). It covers the five multiple-choice subsets that lm-evaluation-harness has as haerae: standard_nomenclature, loan_words, rare_words, general_knowledge, history (1,091 questions). Closes #1448.

• haerae:<subset> uses the same prompt, choices (" (A)" to " (E)") and scoring as lm-eval's haerae.
• haerae_cf:<subset> scores the text of each option appended to the question. The paper describes its evaluation this way (log-likelihood of each option concatenated to the question, run with lm-eval-harness) but doesn't give the prompt, so the "질문: ... 정답:" wording is mine. Which variant should be the default is your call.
• The dataset is HAERAE-HUB/HAE_RAE_BENCH_1.1, pinned to a commit. Its card says cc-by-nc-nd-4.0; lighteval only loads it.
• Not included: the v1.1 subsets lm-eval doesn't have, and the two generative subsets (lyrics and proverbs denoising).
• tests/unit/tasks/test_haerae.py covers the prompt conversion and gold index for both variants.

Data note: 98 of the 176 general_knowledge questions have a literal "### 참고:nan" line (an empty note) inside the question block. haerae (label) keeps the query as is, like lm-eval. haerae_cf drops that line, since its prompt is not lm-eval's anyway; notes with real text are kept.

Validation, with Qwen2.5-0.5B (float32, CPU, 0-shot, chat template off): I compared against lm-eval on a 250-question sample only, not all 1,091. The sample is 50 questions per subset, picked by lighteval's --max-samples (a seeded random subset, not the first 50). I ran lm-eval on the same question ids with --samples. haerae predicts the same choice as lm-eval on all 250. haerae_cf does too on the four subsets without the empty note (200 questions). On general_knowledge, 4 of the 50 sampled questions get a different prediction than with the line kept (31 of the 50 had the line), so haerae_cf no longer matches an lm-eval run that passes the line through; I checked this on the same 250 questions with the same model settings. Accuracy on those 50 went from 0.20 to 0.16, which on a model this small I'd read as noise. lm-eval's HAE_RAE_BENCH and the pinned 1.1 revision have identical query and answer for all 1,091 questions. Prompt and choice tokenization is the same whether encoded separately or together, for all 1,091 questions with the Qwen2.5, polyglot-ko and kogpt2 tokenizers (re-checked with Qwen2.5 after the prompt change).

The Qwen2.5 base tokenizer has a chat template, and lighteval applies it by default. Use override_chat_template=False when you compare log-likelihoods with lm-eval.

AI use: prepared with Claude Code; I reviewed the approach and the results.

[CY-HYUN]

Thanks for opening this, @jhwanseok. I checked the data on the pinned revision (five subsets, 1,091 rows): the 질문/선택지 regex matches every query, every row has five options with an (A) to (E) label, and "### 참고:nan" is in 98 of the 176 general_knowledge rows, as you wrote.

Two small things:

1. haerae.py is added with mode 100755. All 110 other .py files under src/lighteval/tasks/tasks/ are 100644.
2. haerae_cf uses your own prompt, so there is no lm-eval output to match. You could drop the "### 참고:nan" line there and keep the label variant as is for parity. Your call.

[jhwanseok]

Thanks, both done. haerae.py is 100644 now, and haerae_cf drops the "### 참고:nan" line (real notes stay); the label variant is unchanged.

Re-ran haerae_cf with Qwen2.5-0.5B on the same 250 questions: the four subsets without that line give identical predictions, and on general_knowledge 4 of 50 predictions changed (31 of the 50 had the line). Accuracy there went from 0.20 to 0.16, which on a model this small I'd read as noise. I updated the PR description to say haerae_cf no longer matches lm-eval on those.