Latency Budgets
Each workflow gets a speed limit in writing — how slow is too slow — enforced before every release and watched in production, so slowness has a name and an address.
A pizzeria doesn't promise 'fast', it promises a number — and when a delivery takes longer, everyone knows. A latency budget is that promise for an AI workflow: a written limit on how long answers may take, checked before every release and watched in production, so 'it feels slow' becomes a fact you can act on.
The problem, in plain words
A user tells you the assistant feels slow. Slow how? Since when? On which step? Nobody knows, because nobody ever wrote down what fast was supposed to mean. The complaint bounces between teams — the model people say the model is fine, the app people say the app is fine — and it dies in a channel somewhere. Meanwhile a change shipped last month added two seconds to every answer, and no alarm went off, because there was no line to cross.
What we set up
Each workflow declares a latency budget: a p95 and a p99 target (p95 means 95 out of 100 requests must finish within the target time; p99, 99 out of 100 — the point is to measure the slow ones, not the comfortable average). The eval harness (the test bench) measures both on the saved test cases before any change is promoted; a change that blows the budget is blocked at the gate. In production, telemetry (continuous measurement of real traffic) tracks compliance with the same budget, and when it's broken, an alert fires that names the responsible step inside the workflow — not just 'something is slow somewhere'.
How it works, step by step
- Each workflow gets a written speed limit
Targets that cover nearly all requests (the p95 and p99 marks), not just the average day.
- Every change is timed before release
The test bench measures each candidate against the budget. Too slow means no promotion, same as a failed quality test.
- Production is measured continuously
Real traffic, real timings, checked against the same budget the tests used.
- Slowdowns trigger a named alert
Not 'things are slow' but which workflow, and which step inside it — the retrieval, the model call, the tool.
- The budget is renegotiated deliberately
If a workflow legitimately needs more time, the budget changes on purpose, in writing — never by quiet erosion.
What changes for you
Before, 'feels slow' was a complaint nobody could route, and latency crept upward invisibly, one change at a time. After, latency is a number with an owner: regressions are caught at the release gate, and production slowdowns arrive as alerts pointing at a specific step, so the fix starts in minutes instead of in a debate. What it won't do: it won't make anything faster by itself. It makes slowness visible and attributable — which is exactly what lets someone fix it.