Repository navigation
[BUG] LightevalTaskConfig doesn't support comparison, but Cache management uses sorted on it #1024
Description
Describe the bug
Specify two tasks as below:
helm|boolq|0
helm|boolq:contrastset|0
The current logic will assume "helm|boolq" is a group and will add "helm|boolq:contrastset" again, then "helm|boolq:contrastset" will appear twice, later at SampleCache:_get_task_hash sorted is used to sort these two, and
an error "<" is not supported on LightevalTaskConfig is raised
To Reproduce
Specify two tasks as below in yaml:
helm|boolq|0
helm|boolq:contrastset|0
Expected behavior
Run should succeed
Version info
0.11.0
Activity
clefourrier commentedon Oct 21, 2025
Oh cool edge case, good catch! Here we would actually rename
boolqtoboolq:defaultso that a task does not also carry a group name.colinzuo commentedon Oct 21, 2025
How about sorted(self.registry.task_to_configs[f"{task_suite}|{task_name}"]), currently comparison operator is not defined for LightevalTaskConfig, so this place will fail if more than one config is met, any thought?
HarshDubey23 commentedon Oct 24, 2026
Re-checked on current main (
7fa19dc6): thesorted()crash is gone (the cache now sorts config strings), but the expansion issue remains.Registry(tasks="boolq|0")loadsboolq|0andboolq:contrastset|0. The same happens forimdb,bold,lsat_qaandbbq(bbqadds all 11 subsets), because each of these is both a registered task and a superset prefix. Two possible fixes: (a) rename the base tasks toX:default, as you suggested, which changes public task names; or (b) make an exact task match win in_expand_task_definition, which keeps names and stops superset expansion for these five. Which would you accept? I can open a PR with a regression test. (Prepared with AI assistance; I'm not a maintainer.)HarshDubey23 commentedon Oct 24, 2026
Repro on current main (7fa19dc), no model or API needed: running Registry(tasks="boolq|0").load_tasks() returns both boolq|0 and boolq:contrastset|0. The same happens with imdb|0. Requesting boolq|0,boolq:contrastset|0 adds the boolq:contrastset config twice.
Why it matters: asking for one task runs and reports tasks the user didn’t request. For bbq that means the full set plus all 11 subsets, so the same data is evaluated twice.
Affected names (each is both a registered task and a superset prefix): bbq, bold, boolq, imdb, lsat_qa.
With either fix, I’d add a regression test checking that every registered task name expands to exactly itself, and that boolq|0 loads only boolq|0.
Note: on a fresh install with huggingface_hub 2.x, Registry() currently fails earlier with list_models() got an unexpected keyword argument 'model_name' (fixed by #1427). I ran this with huggingface_hub<2.