You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Evaluate a local embedding router on semantic routing challenges #55
Extend the offline routing experiment with semantic paraphrases and near-miss cases, then compare Lucid current deterministic routing with a small local embedding candidate.
Scope:
Keep this experiment-only; no runtime integration.
Add family-isolated semantic and negative challenge cases.
Support an optional local Ollama embedding adapter.
Record accuracy, high-confidence wrong direct actions, abstention/clarification behavior, and latency.
Test installed local embedding models without committing model files or generated reports.
Adoption rule: a local model must improve meaningful semantic coverage without introducing unsafe confident actions or unacceptable latency. If the deterministic router remains sufficient, record that result and do not add runtime complexity.
Implemented and measured in PR #56. The honest de-leaked result is: current 37/46, calibrated Nomic 42/46, calibrated Qwen 39/46. Nomic is smaller and faster, but its 55.6% semantic scope accuracy is not sufficient for runtime adoption, so this PR keeps the model path experiment-only.
Activity
triggeredcode commentedon Oct 20, 2026
Implemented and measured in PR #56. The honest de-leaked result is: current 37/46, calibrated Nomic 42/46, calibrated Qwen 39/46. Nomic is smaller and faster, but its 55.6% semantic scope accuracy is not sufficient for runtime adoption, so this PR keeps the model path experiment-only.
feat(evals): compare local semantic routing (#55) (#56)