[Cookie0103]

What changes

Allow SampleCache to persist regenerated responses when the previous Parquet row cannot be decoded as a ModelResponse.

Startup indexing already skips undecodable rows so those docs will be recomputed. However, cache_samples builds its duplicate-ID set from every old row, keeps the invalid one, and discards the replacement. The @cached wrapper then reloads the same invalid row and raises. In a legacy-schema file, Arrow can also carry the obsolete response field into newly appended rows.

Validate old rows with the same _load_sample criterion before merging. Discard rejected rows with a count warning, retain valid cached work and its existing precedence, and append regenerated responses. Cache keys, layout, public APIs and validation criteria are unchanged.

Example: a cache containing a valid response for good and a null response for bad, followed by a request for [new, good, bad], now computes only [new, bad], returns all three responses in request order, and reuses all three after reopening. An obsolete response-schema field is covered separately.

Tests

Real local Parquet files and the real @cached wrapper, with a scripted model body; no provider credentials or GPU inference.

HF_HOME=/tmp/lighteval-test-hf HF_HUB_OFFLINE=1 HF_DATASETS_OFFLINE=1 \
  .venv/bin/python -m pytest tests/unit/utils/test_cache_recovery.py -q

• Original commit b97a6533bec22c4184b88526366fdf5d73d502c9: 6 failed, 1 passed. The failures reproduce null/obsolete-schema response loading, across GENERATIVE, LOGPROBS and PERPLEXITY.
• Fixed: 7 passed; covers recomputation, preservation of valid outputs, returned order, persisted rows, reopening, and valid duplicate precedence.
• Related offline selection (test_cache_recovery.py, test_utils.py, test_model_input.py, test_cache_directory_structure, test_cache_sglang): 18 passed, 9 subtests passed separately.
• Repository-pinned pre-commit run --all-files and hooks on both changed files: passed. git diff --check: passed.
• A broader local selection also attempted cache/model/pipeline tests: 21 failed, 21 passed, 1 skipped, 36 subtests passed. Running the same existing tests against the unchanged base reproduced the same 21 failures, with 14 passed, 1 skipped, 36 subtests passed. These are unavailable offline tokenizers, optional vLLM support, and default-cache sandbox permission failures; the seven new passing cases explain the pass-count difference. This is not a full-suite success claim.

Environment: macOS ARM64, Python 3.12.13, editable .[dev] installation. Concurrent writers, atomic file replacement, cache identity changes and stronger response validation are outside this repair.

AI assistance: OpenAI Codex investigated, implemented, tested and performed a separate AI review. The account owner has not personally reviewed or run these tests.