Cache-aware Routing

Route between models at natural cache boundaries rather than mid-conversation, preserving prompt cache efficiency while still allowing model switching when beneficial.

Why It Matters

Switching models mid-conversation breaks the prompt cache because the new model sees a different token prefix. The cache miss can cost more than the routing change saves. Cache-aware routing avoids this by switching only when the cache would be lost anyway.

Natural Cache Boundaries

• First turn: no cache exists yet, so any model choice is free.
• After compaction: the prompt prefix resets when the transcript is summarized, so the cache is already lost.
• After a long idle period: many providers evict cached prefixes after a timeout.

Between these points, the selected model stays in place so the cache can keep building.

Trade-offs

• Pros: preserves cache hit rate; avoids hidden costs from frequent model switching.
• Cons: slightly less flexible than per-turn routing; requires the harness to track cache state.

See Also

• Prompt Caching — the mechanism that cache-aware routing is designed to preserve.
• Model Routing — general model routing strategies.
• Compaction — compaction is a natural cache boundary where routing can safely switch models.
• Provider Pinning — the same problem below the model layer: an aggregator rerouting to another backend also loses the cache.

Tools

• GitHub Copilot Auto model selection — routes at natural cache boundaries to avoid breaking the prompt cache.