Batch Inference
Submitting large sets of model requests through a provider's asynchronous batch API — typically at about half the price of interactive calls — for AI workloads that don't need an immediate answer.
Production AI is a system: context, tools, permissions, traces, evals, and feedback loops around the model.
What it is
Anthropic's Message Batches API and OpenAI's Batch API accept a set of requests and return results within a processing window (minutes to 24 hours) at roughly a 50% discount. The trade is latency for price — a good trade for classification sweeps, extraction backfills, re-embedding, and nightly enrichment pipelines.
Why it matters
A large share of production AI work is background work. Paying interactive prices for a pipeline that runs at 3 a.m. wastes budget the workload never needed to spend.
How we use it
Background pipelines in our stack submit through a batch seam and collect results in a second pass. Usage logging attributes batch cost the same way as interactive cost, so the saving shows up in the cost report instead of being assumed.
Related resources
The total spend on language-model APIs across an organization — input tokens, output tokens, embeddings, fine-tuning — and the practice of attributing, optimizing, and budgeting it.
Reusing the already-processed prefix of a prompt across calls — system instructions, tool definitions, long documents — so repeated context costs a fraction of full price and returns faster.
How an AI system decides which model to call for each step — based on privacy, cost, latency, quality, and what happens when a provider goes down.