Provider Pinning

Force every request of a conversation to the same backend inference provider, so the prompt cache built on the previous turn is still there on the next one.

Why It Matters

Aggregators such as OpenRouter serve one model name from several backends (Together, Fireworks, DeepInfra, Baseten, the first-party API...). The router picks a backend per request, optimizing for price, latency or availability — and it may pick a different one on the very next turn.

Prompt caches are not shared across backends. Each provider keeps its own KV cache, keyed by its own prefix hash. So a silent backend switch mid-conversation means:

the cached prefix is gone,
the whole prompt is billed at full uncached input price,
and time-to-first-token grows with prefix length.

In an agentic loop where the system prompt, tool schemas and transcript can be tens of thousands of tokens re-sent every turn, a cache miss is typically 5–10× the cost of a cache hit on that prefix. One unlucky reroute per five turns can erase the entire saving of Prompt Caching.

How to Pin

On OpenRouter, pass a provider block that names a single backend and forbids fallback:

{
  "model": "...",
  "provider": {
    "order": ["fireworks"],
    "allow_fallbacks": false
  },
  "messages": [...]
}

Related knobs worth setting alongside it:

provider.require_parameters — drop backends that silently ignore cache_control.
Sticky routing by session id, where the aggregator offers it.
Pin the exact model revision too: a quantization or version change is also a new cache.

Going direct to a first-party API is the degenerate case of provider pinning: one backend, always.

Trade-offs

Pros: cache hit rate becomes predictable; cost per turn stops fluctuating; latency is stable.
Cons: no automatic failover — if the pinned backend is down or rate-limited, the request fails instead of being rerouted. Availability must be handled explicitly (retry, then re-pin), and the re-pin is itself a cold start.
Pinning to a cheap-but-slow backend can cost more in wall-clock than it saves in tokens.

Detecting the Problem

Aggregator responses report which backend served the request and how many tokens were cache reads. Log both. A conversation where the backend field changes between turns, or where cache_read_input_tokens collapses to zero on a turn that should have hit, is an unpinned reroute.

Tools