[EvanZhuang]

with use_chat_template, lighteval renders the template to text (apply_chat_template(..., tokenize=False), which already has <|begin_of_text|>) and then tokenizes it with add_special_tokens=True, so Llama 3, Gemma 2/3, and Mistral v0.3 chat prompts start with two BOS tokens ([128000, 128000, 128006, ...]). huggingface/open-r1#462 noted this earlier and #598 made add_special_tokens=False settable from the CLI while keeping the True default (setting it globally strips the BOS that plain-text base prompts need).

lighteval vllm on meta-llama/Llama-3.2-1B-Instruct (gsm8k|0,ifeval|0, greedy, 1x H100):

Metric	main	This PR	add_special_tokens=False
First 2 prompt IDs (n=1,860)	[128000, 128000]	[128000, 128006]	[128000, 128006]
GSM8K extractive_match (n=1,319)	35.94	46.32	46.25
IFEval prompt-strict (n=541)	44.55	49.72	49.54
IFEval inst-strict (n=834)	56.95	62.71	62.71

Fix: in VLLMModel, TransformersModel, and SGLangModel, add_special_tokens returns self._add_special_tokens and not self.use_chat_template, skipping extra special tokens only for rendered chat templates while base mode keeps its BOS. If a chat template omits BOS while its tokenizer adds one, chat mode now matches apply_chat_template(tokenize=True) and omits that BOS.

Tests: unit tests in test_vllm_model.py and test_transformers_model.py (Llama-3.2-1B-Instruct chat/base, Qwen3-0.6B), and override_chat_template=False in test_loglikelihood_rolling_matches_manual_scoring.

repro (prints the first prompt token ids): https://gist.github.com/EvanZhuang/90f4eba2e660cfdd1997ea77f129191b