Prompt Caching

Cache stable prompt prefixes (system prompts, large reference files, repository context) so repeated tokens are not re-processed or re-billed on every request.

How It Works

Inference servers or APIs detect that the beginning of a new request matches a previously seen prefix. They reuse the precomputed key-value attention state from the cache instead of recomputing it, reducing latency and cost.

What to Cache

• system prompts
• large reference documents
• repository context that does not change between turns
• tool definitions (when not using dynamic loading)

Limitations

• Caching is prefix-based: any change to the cached portion invalidates the cache.
• Every provider enforces a minimum cacheable prompt size, below which nothing is cached regardless of prefix stability — a zero-cache-hit response there is expected behavior, not a caching failure. Measured minimums vary by model family, not just by vendor:
• GPT-5.6-class models: ~1,024 tokens.
• Anthropic Claude Sonnet / Opus (claude-sonnet-completions.py, claude-sonnet-5-completions.py): ~1,024 tokens.
• Anthropic Claude Haiku (claude-haiku-completions.py, model claude-haiku-4-5-20251001): ~4,096 tokens — 4x higher than Sonnet/Opus. Empirically verified: prompts totalling ≤4,091 input tokens never produce cache_creation_input_tokens or cache_read_input_tokens, even on repeated identical calls; at ≥4,092 tokens caching works normally (write on the 1st call, read on repeats).
• Always verify per-model, not per-vendor: model family (Haiku vs. Sonnet/Opus) changes the floor even within one provider.
• Compaction and mid-conversation model switches usually break the cache.
• Provider support varies; some charge a reduced rate for cached tokens rather than eliminating cost entirely.
• Cache entries expire after a provider-set TTL; a request arriving after expiry pays full price again. The longer the TTL, the wider the reuse window and the more of your traffic actually hits cache — so savings scale with TTL length, not just with prefix stability. Some providers let you pay a higher write cost for a longer TTL, which pays off when request spacing regularly exceeds the default window (e.g. bursty or long-idle agent loops).

See Also

• Cache-aware Routing — routing strategy that preserves prompt cache efficiency.
• Compaction — compaction usually breaks the prompt cache.
• Token-Aware Formatting — deduplicating repeated instructions is a complement to caching.
• Provider Pinning — caches are per-backend; behind an aggregator, the cache only survives if the backend does.

Tools

• OpenAI prompt caching — API-level caching for repeated prompt prefixes.
• Anthropic prompt caching — caches large context blocks across API calls.