Skip to content

Evaluate small local decision model - #58

Merged
triggeredcode merged 1 commit into
mainfrom
feat/local-decision-model-experiment
Oct 20, 2026
Merged

triggeredcode merged 1 commit into
mainfrom
feat/local-decision-model-experiment

Conversation

@triggeredcode

Copy link
Copy Markdown
Owner

Closes #57

What changed

  • adds an experiment-only Ollama decision-model routing candidate
  • preserves deterministic and explicit-state routes
  • records calibrated confidence and model evidence in reports
  • documents the Gemma 3 1B calibration result

Result

Gemma 3 1B was eliminated on calibration: 12/16 routes, 50.0% semantic scope accuracy, and 0.0% vague near-miss safety. Wrong answers were reported at 95% confidence, so the holdout was intentionally not opened. Product runtime remains unchanged.

Validation

  • pnpm format:check
  • pnpm lint
  • pnpm typecheck
  • pnpm test (1531 passed, 22 skipped)

@triggeredcode
triggeredcode merged commit 52c4bf9 into main Oct 20, 2026
4 checks passed
@triggeredcode
triggeredcode deleted the feat/local-decision-model-experiment branch October 20, 2026 10:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Evaluate a small local decision model on routing dataset v2

1 participant