Repository navigation
Fill in the translation literals for Latvian, Lithuanian, Maltese and Slovenian - #1473
Open
inchang-ing wants to merge 1 commit into
Open
inchang-ing wants to merge 1 commit into
inchang-ing wants to merge 1 commit into
Conversation
The Belebele cf tasks for these four languages could not be built: prompt construction reads `question_word` and `answer`, and both were left unset on their TranslationLiterals entries, so the documents failed with AttributeError before any model was loaded. Fill in the literal set described in docs/source/contributing-to-multilingual-evaluations.mdx. The punctuation defaults already match these languages, so only the words are set. A parametrized test checks the cf prompt labels for each of the four languages. Fixes huggingface#1471
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this changes
belebele_{lvs,lit,mlt,slv}_Latn_cfcould not be built. Prompt construction for the cf formulation readsquestion_wordandanswer, and theTranslationLiteralsentries for Latvian, Lithuanian, Maltese and Slovenian left every field unset, so the documents failed before any model was loaded:This fills in the literal set described in
docs/source/contributing-to-multilingual-evaluations.mdxfor the four languages. The punctuation defaults (.,,,?,!,:) already match these languages, so only the words are set, and the choice indices stayA/B/C/D.Verification
Building the cf prompt directly, which needs neither a dataset download nor a model, on the four languages from the report.
Before:
After:
The new test
test_multichoice_prompt_cf_uses_language_literalspins the rendered query, the unconditioned query and the choices for the four languages. With the literal entries reverted it fails on all four parameters with theAttributeErrorabove; with them it passes.pytest tests/unit/tasks→ 36 passedruff format --checkandruff checkclean (ruff 0.17.3)Note on the words
These are the standard single-word forms for each language. If you prefer different phrasings for
confirmation_word("vai ne", "ar ne", "hux hekk", "kajne") orneither, they are a one-line change each.Fixes #1471
This change was prepared with AI assistance (DeepSeek). The reproduction, the fix and the tests above were run locally against this branch.