Skip to content

Fill in the translation literals for Latvian, Lithuanian, Maltese and Slovenian - #1473

Open
inchang-ing wants to merge 1 commit into
huggingface:mainfrom
inchang-ing:fix/translation-literals-lv-lt-mt-sl
Open

inchang-ing wants to merge 1 commit into
huggingface:mainfrom
inchang-ing:fix/translation-literals-lv-lt-mt-sl

Conversation

@inchang-ing

Copy link
Copy Markdown

What this changes

belebele_{lvs,lit,mlt,slv}_Latn_cf could not be built. Prompt construction for the cf formulation reads question_word and answer, and the TranslationLiterals entries for Latvian, Lithuanian, Maltese and Slovenian left every field unset, so the documents failed before any model was loaded:

AttributeError: Translation for 'answer' is needed for Language.LATVIAN.

This fills in the literal set described in docs/source/contributing-to-multilingual-evaluations.mdx for the four languages. The punctuation defaults (., ,, ?, !, :) already match these languages, so only the words are set, and the choice indices stay A/B/C/D.

Verification

Building the cf prompt directly, which needs neither a dataset download nor a model, on the four languages from the report.

Before:

belebele_eng_cf: OK  'France is a country in Europe.\nQuestion: What is the capital of France?\nAnswer:'
belebele_lav_cf: FAILED: Translation for 'answer' is needed for Language.LATVIAN.
belebele_lit_cf: FAILED: Translation for 'answer' is needed for Language.LITHUANIAN.
belebele_mlt_cf: FAILED: Translation for 'answer' is needed for Language.MALTESE.
belebele_slv_cf: FAILED: Translation for 'answer' is needed for Language.SLOVENIAN.

After:

belebele_lav_cf: OK  'France is a country in Europe.\nJautājums: What is the capital of France?\nAtbilde:'
belebele_lit_cf: OK  'France is a country in Europe.\nKlausimas: What is the capital of France?\nAtsakymas:'
belebele_mlt_cf: OK  'France is a country in Europe.\nMistoqsija: What is the capital of France?\nTweġiba:'
belebele_slv_cf: OK  'France is a country in Europe.\nVprašanje: What is the capital of France?\nOdgovor:'

The new test test_multichoice_prompt_cf_uses_language_literals pins the rendered query, the unconditioned query and the choices for the four languages. With the literal entries reverted it fails on all four parameters with the AttributeError above; with them it passes.

  • pytest tests/unit/tasks → 36 passed
  • ruff format --check and ruff check clean (ruff 0.17.3)

Note on the words

These are the standard single-word forms for each language. If you prefer different phrasings for confirmation_word ("vai ne", "ar ne", "hux hekk", "kajne") or neither, they are a one-line change each.

Fixes #1471


This change was prepared with AI assistance (DeepSeek). The reproduction, the fix and the tests above were run locally against this branch.

The Belebele cf tasks for these four languages could not be built: prompt
construction reads `question_word` and `answer`, and both were left unset on
their TranslationLiterals entries, so the documents failed with
AttributeError before any model was loaded.

Fill in the literal set described in
docs/source/contributing-to-multilingual-evaluations.mdx. The punctuation
defaults already match these languages, so only the words are set.

A parametrized test checks the cf prompt labels for each of the four
languages.

Fixes huggingface#1471
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[BUG] Belebele cf tasks fail for Latvian, Lithuanian, Maltese and Slovenian: missing translation literals

1 participant