Prompt Caching
Reusing the already-processed prefix of a prompt across calls — system instructions, tool definitions, long documents — so repeated context costs a fraction of full price and returns faster.
Production AI is a system: context, tools, permissions, traces, evals, and feedback loops around the model.
What it is
Providers cache the computed state of a prompt prefix so subsequent calls that share it skip recomputation. Anthropic exposes explicit cache-control breakpoints with a TTL, with cache reads priced at roughly a tenth of input price; OpenAI applies caching automatically on repeated prefixes. Either way, the stable part of the prompt stops being billed like new tokens.
Why it matters
Agents resend the same system prompt, personality, and tool definitions on every turn. On long sessions that repeated prefix dominates spend and latency. With caching, the marginal turn costs mostly what is actually new in it.
How we use it
Cache breakpoints sit at the stable prefix of agent prompts in our stack, with the TTL configured per workload. Usage logs record cache reads and cache writes as their own token classes, so the saving is measured in the cost report rather than assumed.
Related resources
The total spend on language-model APIs across an organization — input tokens, output tokens, embeddings, fine-tuning — and the practice of attributing, optimizing, and budgeting it.
The total number of tokens an AI agent has available for instructions, memory, retrieved context, conversation history, and tool results — and how that budget is allocated across them.
The discipline of deciding what an AI model sees on every call — instructions, retrieved data, memory, tool definitions, examples — and how to assemble them reliably as the workflow grows.
Submitting large sets of model requests through a provider's asynchronous batch API — typically at about half the price of interactive calls — for AI workloads that don't need an immediate answer.