Skip to content

Evaluate a small local decision model on routing dataset v2 #57

Closed

Description

@triggeredcode

Compare the installed gemma3:1b local model with the current heuristic and Nomic embedding result on the frozen routing dataset v2.

Scope:

  • Experiment-only Ollama adapter; no product runtime integration or model download.
  • Preserve exact meta commands, deterministic edits, lane selection, and all product safeguards.
  • Ask the model only for otherwise-unknown scope classification.
  • Record semantic and near-miss accuracy, abstention, cold/warm latency, and high-confidence wrong direct actions.
  • Use family-isolated splits and do not tune on holdout.

Decision rule: keep the result as evidence. Do not adopt unless it materially beats Nomic while preserving 100% vague near-miss safety and acceptable latency/model size.

Activity

  1. triggeredcode commented on Oct 20, 2026

    @triggeredcode
    OwnerAuthor

    Implemented in #58. Gemma 3 1B was evaluated behind the existing deterministic safety boundary and rejected on calibration: 12/16 routes, 50.0% semantic scope accuracy, 0.0% vague near-miss safety, with wrong answers at 95% confidence. The holdout stayed closed and no product runtime behavior changed. Full validation is green.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

No labels
No labels

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions