Repository navigation
Pin each bot to the OpenRouter upstream that served its last call - #305
Merged
Merged
Conversation
OpenRouter spreads a model's calls across its upstreams (34 for glm-5.3-flash), each with its own prompt cache, so most runs cached only a fixed ~40k prefix while their prompts doubled. Unless openrouter_provider_order is set explicitly, record which upstream answered each call and route the bot's next call there (fallbacks on, so the pin follows when that upstream fails). - OpenRouterChatOpenAI keeps OpenRouter's `provider` field in response_metadata - UpstreamPins maps display names to slugs via the model's endpoint list (guessing when offline) - StickyUpstreamMiddleware applies the pin per call, preserving the model's own extra_body - the per-call log line now names the upstream Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
time.monotonic() starts near zero at boot, so the 'fetched at 0' default looked fresh for the first ten minutes of uptime and every slug was guessed (caught by CI on a fresh runner). Lock per model so a slow fetch only holds up that model's runs; fetch timeout 10s -> 5s. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
regnull
added a commit
that referenced
this pull request
Oct 18, 2026
* Skip OpenRouter upstreams that do not cache Pinning kept bots on whichever upstream OpenRouter picked first, and its price-weighted default picks cheap upstreams that barely cache: after #305 bots sat on OpenInference, Venice and PrimeIntellect with ~2% of prompt tokens cached. Judge warm calls (same upstream as the run's previous call): three in a row with under 25% cached skip that upstream for the model for 6h via provider.ignore and unpin its bots, so they settle on an upstream that caches. Capped at 10 skips per model; skips lapse. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Never skip the last upstream; judge warm calls on what could be cached, per bot Review fixes for the cache learning: - Only skip an upstream when the model's endpoint list is known and another upstream remains, so the ignore list can never cover every upstream and fail all calls with 'no endpoints'. - Judge a warm call against min(this prompt, previous prompt): a big new tool result can only be cached from the call after it, so it no longer counts as a miss. - Miss streaks are per bot, so one bot's hit does not hide another's misses and ordinary misses from several bots do not add up to a skip. - A pin to a skipped upstream (re-pinned by a call in flight) is dropped from the request. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
OpenRouter serves
z-ai/glm-5.3-flashfrom 34 upstream endpoints and, with noprovider.order, spreads calls across them. Each upstream has its own prompt cache, so a call that lands somewhere new starts cold. In the logs most runs cached a fixed prefix (e.g. exactly 41,472 tokens on 81 of 87 calls) while the prompt grew from 52k to 102k. Only 47% of the engineer's prompt tokens were cached.What
Unless
openrouter_provider_orderis set (an explicit choice still wins), each bot's next OpenRouter call is routed to the upstream that served its previous call:OpenRouterChatOpenAI: aChatOpenAIsubclass that keeps OpenRouter's top-levelproviderfield, which langchain-openai drops. It's stored as a one-item list because streamed metadata merges strings by concatenation. Tagged only on closing chunks.UpstreamPins: per-(bot, model) pins. Maps display names ("Z.AI") to routing slugs (z-ai) via the model's public endpoint list, cached for 6h; a guessed slug is the fallback.StickyUpstreamMiddleware: setsextra_body.provider = {order: [slug], allow_fallbacks: true}per call, keeping the model's ownextra_body(e.g. the web plugin). When a fallback answers, the pin moves there.upstream=…. The setting's description and the README explain the default.Side tasks (renaming, memory reflection) aren't pinned.
Tests
tests/test_upstream.py: recording from streamed and non-streamed replies, the factory, name→slug mapping and fallback, pinning plus fallback-follow plus extra_body merge, and explicit order winning.Not yet seen against live OpenRouter. If streamed chunks lack
provider, the log showsupstream=-and nothing is pinned (no breakage).🤖 Generated with Claude Code