Q: How do I calculate my cache hit rate?

Cache hit rate = (calls that can use the cache) / (total calls). A call uses the cache if: (1) The cached prefix hasn't expired (5-minute TTL for Anthropic ephemeral cache). (2) The prefix is byte-for-byte identical to what was cached. For high-volume applications (>10 calls/minute), near-100% cache hit rate is achievable for the stable portion of your prompt. For low-volume applications (<1 call/5 minutes), the cache frequently expires and hit rates are much lower. This calculator models the hit rate based on your calls-per-minute and cache TTL.

Q: How much can prompt caching save on a large system prompt?

Example: Claude 3.5 Sonnet, 8,000-token system prompt, $3/M standard input rate. Without caching: 8,000 × $3/1,000,000 = $0.024 per call for the system prompt alone. With caching (90% hit rate): 10% of calls pay cache write ($3.75/M × 8,000 = $0.030); 90% pay cache read ($0.30/M × 8,000 = $0.0024). Blended cost per call = 0.10 × $0.030 + 0.90 × $0.0024 = $0.003 + $0.00216 = $0.00516. Savings vs uncached: $0.024 - $0.00516 = $0.01884 per call. At 10,000 calls/day: $188.40/day savings = $5,652/month.

Q: Does prompt caching affect response quality?

No. The cached KV values are mathematically equivalent to recomputing them from scratch — the model outputs identical distributions whether it reads from cache or recomputes. There is no quality degradation from caching. The only behavioral difference is latency: cache hits are faster (skip the input processing for the cached prefix) and cache writes are slightly slower (compute + store). For streaming applications, the time-to-first-token is faster on cache hits.

Q: How does Anthropic's caching compare to OpenAI's?

Anthropic (as of mid-2024): cache_control blocks in message content; 1,024 token minimum; 5-minute TTL; cache write = 125% of standard; cache read = 10% of standard. OpenAI (as of November 2024): automatic prompt caching for prompts 1,024+ tokens (no explicit control needed); 50% discount on cached input tokens; caches the longest prefix that is cached and matches the new request. Key difference: Anthropic requires explicit cache_control markers (giving developers precise control); OpenAI caches automatically (simpler but less control). Both offer substantial savings for appropriate workloads.

Question 1

What is prompt caching and how does it work technically?

Accepted Answer

When you make an LLM API call, the model must process (attend over) all input tokens to compute the key-value (KV) attention cache. Prompt caching stores this pre-computed cache on the provider's infrastructure so subsequent calls with the same prefix can skip the recomputation. Anthropic's implementation: add a cache_control: {type: 'ephemeral'} block to the content you want cached. The cache lasts 5 minutes (ephemeral) and resets the timer on each access. Cache writes cost 125% of standard input rate; cache reads cost 10% of standard rate. The minimum cacheable block is 1,024 tokens.

Question 2

What types of applications benefit most from prompt caching?

Accepted Answer

Applications with large, stable context that repeats across calls: (1) Customer service bots with large system prompts (product catalogs, policies, procedures). (2) Document Q&A where the document is large and the same document is queried multiple times. (3) Code assistants with large codebases injected into context. (4) Multi-turn conversations where conversation history grows but the system prompt is fixed. (5) RAG applications where the same retrieved chunks appear across multiple queries. Applications that don't benefit: one-off single calls, highly personalized prompts that change every call, short system prompts (<1,024 tokens).

Question 3

How do I calculate my cache hit rate?

Accepted Answer

Cache hit rate = (calls that can use the cache) / (total calls). A call uses the cache if: (1) The cached prefix hasn't expired (5-minute TTL for Anthropic ephemeral cache). (2) The prefix is byte-for-byte identical to what was cached. For high-volume applications (>10 calls/minute), near-100% cache hit rate is achievable for the stable portion of your prompt. For low-volume applications (<1 call/5 minutes), the cache frequently expires and hit rates are much lower. This calculator models the hit rate based on your calls-per-minute and cache TTL.

Question 4

How much can prompt caching save on a large system prompt?

Accepted Answer

Example: Claude 3.5 Sonnet, 8,000-token system prompt, $3/M standard input rate. Without caching: 8,000 × $3/1,000,000 = $0.024 per call for the system prompt alone. With caching (90% hit rate): 10% of calls pay cache write ($3.75/M × 8,000 = $0.030); 90% pay cache read ($0.30/M × 8,000 = $0.0024). Blended cost per call = 0.10 × $0.030 + 0.90 × $0.0024 = $0.003 + $0.00216 = $0.00516. Savings vs uncached: $0.024 - $0.00516 = $0.01884 per call. At 10,000 calls/day: $188.40/day savings = $5,652/month.

Question 5

Does prompt caching affect response quality?

Accepted Answer

No. The cached KV values are mathematically equivalent to recomputing them from scratch — the model outputs identical distributions whether it reads from cache or recomputes. There is no quality degradation from caching. The only behavioral difference is latency: cache hits are faster (skip the input processing for the cached prefix) and cache writes are slightly slower (compute + store). For streaming applications, the time-to-first-token is faster on cache hits.

Question 6

How does Anthropic's caching compare to OpenAI's?

Accepted Answer

Anthropic (as of mid-2024): cache_control blocks in message content; 1,024 token minimum; 5-minute TTL; cache write = 125% of standard; cache read = 10% of standard. OpenAI (as of November 2024): automatic prompt caching for prompts 1,024+ tokens (no explicit control needed); 50% discount on cached input tokens; caches the longest prefix that is cached and matches the new request. Key difference: Anthropic requires explicit cache_control markers (giving developers precise control); OpenAI caches automatically (simpler but less control). Both offer substantial savings for appropriate workloads.

Prompt Caching Savings Calculator

Inputs coming in next batch

Frequently asked questions