Concepts
Caching
Prefix caching is automatic. There is nothing to turn on and no cache-control markup.
When the start of a request matches the start of a recent request to the same backend, the backend reuses its work for the matching tokens. Those are cached tokens.
Billing
Cached tokens are billed at the model's cached_input price instead of its input price. For most models that is about 10% of the input price; see pricing.cached_input in GET /v1/models for each model's exact rate.
cost = (input_tokens − cached_tokens) × input_price
+ cached_tokens × cached_input_price
+ output_tokens × output_pricePrices are per 1M tokens. Batch requests then take 50% off the total.
Seeing cache hits
| Where | Field |
|---|---|
| Response header | x-utmost-cached-tokens (not on streams) |
| Chat completions | usage.prompt_tokens_details.cached_tokens |
| Responses | usage.input_tokens_details.cached_tokens |
| Messages | usage.cache_read_input_tokens |
prompt_tokens and x-utmost-input-tokens include cached tokens. In Messages, input_tokens excludes them, following the Anthropic convention.
We report the cached count the backend returns. If a backend reports none, cached tokens are 0 and you pay the full input price.
Getting more hits
Caches match from the first token. A change early in the prompt makes everything after it miss.
- Put stable content first: system prompt, tool definitions, reference documents, few-shot examples.
- Put what changes last: the user's question, retrieved snippets, timestamps.
- Keep the stable part byte-identical between requests. Do not put a date, request id or random ordering in the system prompt.
- Keep tool definitions in a fixed order.
- In a conversation, append turns. Do not rewrite earlier ones.
- Pin a model for workloads that depend on the cache. Each model, and each deployment of it, has its own cache. With an auto alias, requests can land on different models, and a fallback moves a request to another deployment.
Cache entries expire after a period that depends on the backend and its load. There is no guaranteed lifetime.