You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Evaluate a small local decision model on routing dataset v2 #57
Compare the installed gemma3:1b local model with the current heuristic and Nomic embedding result on the frozen routing dataset v2.
Scope:
Experiment-only Ollama adapter; no product runtime integration or model download.
Preserve exact meta commands, deterministic edits, lane selection, and all product safeguards.
Ask the model only for otherwise-unknown scope classification.
Record semantic and near-miss accuracy, abstention, cold/warm latency, and high-confidence wrong direct actions.
Use family-isolated splits and do not tune on holdout.
Decision rule: keep the result as evidence. Do not adopt unless it materially beats Nomic while preserving 100% vague near-miss safety and acceptable latency/model size.
Implemented in #58. Gemma 3 1B was evaluated behind the existing deterministic safety boundary and rejected on calibration: 12/16 routes, 50.0% semantic scope accuracy, 0.0% vague near-miss safety, with wrong answers at 95% confidence. The holdout stayed closed and no product runtime behavior changed. Full validation is green.
Activity
triggeredcode commentedon Oct 20, 2026
Implemented in #58. Gemma 3 1B was evaluated behind the existing deterministic safety boundary and rejected on calibration: 12/16 routes, 50.0% semantic scope accuracy, 0.0% vague near-miss safety, with wrong answers at 95% confidence. The holdout stayed closed and no product runtime behavior changed. Full validation is green.
feat(evals): evaluate local decision model (#57) (#58)