Inference
Running a trained model to get an answer — the thing you pay for per token, and the layer where latency, cost, and deployment choices live.
Production AI is a system: context, tools, permissions, traces, evals, and feedback loops around the model.
What it is
Training builds a model; inference uses it. Every chat reply, extraction, embedding, and agent step is an inference call. For almost every organization the practical question is not how models are trained but where and how inference runs: a provider's API, a cloud platform, or your own hardware.
Why it matters
Inference is the recurring cost of AI in production, priced per token. Its latency shapes the user experience, and where it runs determines which data can flow through it — the reason regulated workloads sometimes require private inference.
How it works
Most workloads run on provider APIs (Anthropic, OpenAI, Google) behind a gateway that routes by cost, quality, and privacy class. Open-weight models served by engines like vLLM or Ollama cover private or air-gapped requirements. Prompt caching and batch APIs are the two standard levers that cut inference spend without touching quality.
Related resources
How an AI system decides which model to call for each step — based on privacy, cost, latency, quality, and what happens when a provider goes down.
AI models running on machines you control — inside your own walls or your private cloud — so sensitive data gets AI help without ever leaving the building.
Reusing the already-processed prefix of a prompt across calls — system instructions, tool definitions, long documents — so repeated context costs a fraction of full price and returns faster.
Submitting large sets of model requests through a provider's asynchronous batch API — typically at about half the price of interactive calls — for AI workloads that don't need an immediate answer.