Skip to content

Evaluate local semantic routing without runtime integration - #56

Merged
triggeredcode merged 1 commit into
mainfrom
feat/semantic-routing-experiment
Oct 20, 2026
Merged

triggeredcode merged 1 commit into
mainfrom
feat/semantic-routing-experiment

Conversation

@triggeredcode

Copy link
Copy Markdown
Owner

Closes #55

This expands the offline experiment to dataset v2 and records a local embedding comparison. It does not add a model or routing dependency to Lucid runtime.

Measured result on Apple M4 Pro:

  • Current: 37/46; semantic scope 0%; vague near-miss scope 100%
  • Nomic Embed Text, calibrated margin 0.04: 42/46; semantic 55.6%; near-miss 100%; challenge p50/p95 9.6/11.5 ms; cold max 175 ms
  • Qwen3 Embedding 0.6B, same margin: 39/46; semantic 22.2%; near-miss 100%; challenge p50/p95 24.7/34.3 ms; cold max 415 ms
  • All candidates kept lane/action accuracy at 100% and produced zero high-confidence wrong direct actions

Decision: Nomic is the leading local candidate, but 55.6% semantic scope accuracy is not enough to justify product integration. Keep it experiment-only.

Validation:

  • pnpm format:check
  • pnpm lint
  • pnpm typecheck
  • pnpm test (113 files passed, 5 skipped; 1529 tests passed, 22 skipped)
  • Docdex import diagnostics clean

@triggeredcode
triggeredcode merged commit c90c676 into main Oct 20, 2026
4 checks passed
@triggeredcode
triggeredcode deleted the feat/semantic-routing-experiment branch October 20, 2026 05:58
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Evaluate a local embedding router on semantic routing challenges

1 participant