Repository navigation
Skip OpenRouter upstreams that do not cache - #307
Merged
Merged
Conversation
Pinning kept bots on whichever upstream OpenRouter picked first, and its price-weighted default picks cheap upstreams that barely cache: after #305 bots sat on OpenInference, Venice and PrimeIntellect with ~2% of prompt tokens cached. Judge warm calls (same upstream as the run's previous call): three in a row with under 25% cached skip that upstream for the model for 6h via provider.ignore and unpin its bots, so they settle on an upstream that caches. Capped at 10 skips per model; skips lapse. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…d, per bot Review fixes for the cache learning: - Only skip an upstream when the model's endpoint list is known and another upstream remains, so the ignore list can never cover every upstream and fail all calls with 'no endpoints'. - Judge a warm call against min(this prompt, previous prompt): a big new tool result can only be cached from the call after it, so it no longer counts as a miss. - Miss streaks are per bot, so one bot's hit does not hide another's misses and ordinary misses from several bots do not add up to a skip. - A pin to a skipped upstream (re-pinned by a call in flight) is dropped from the request. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
#305 pins each bot to the upstream that served its last call, but OpenRouter's price-weighted default routing picks cheap upstreams first, and those barely cache. Since that deploy, 123 calls sat on OpenInference, Venice and PrimeIntellect with 1.7% of prompt tokens cached (avg 22.9k prompt, 0.4k cached).
What
UpstreamPinsnow learns which upstreams cache:provider.ignorealongside the pin'sorder/allow_fallbacks, so OpenRouter picks elsewhere and the next pin lands on a different upstream.openrouter_provider_orderstill bypasses everything.Tests
tests/test_upstream.py: miss streak and reset, skip/unpin/expiry, the 10-skip cap, the full middleware sequence (pinned → 3 warm misses →ignore→ repinned to a caching upstream), and switched calls not being judged.🤖 Generated with Claude Code