Trace Replay
Every agent run is recorded in enough detail to replay it exactly — so 'why did the AI do that?' becomes a question with an answer instead of a shrug.
Think of a flight recorder. After something goes wrong in the air, investigators don't guess — they replay exactly what the instruments saw and what the systems did. Trace replay is a flight recorder for your AI workflows: last Thursday's odd answer can be re-run today, exactly as it happened.
The problem, in plain words
A user reports that the agent gave a bizarre answer last Thursday. You try the same question today — it works fine. Now what? The documents have changed since Thursday, the model's answers vary from run to run, and the tools return different data than they did that day. So 'cannot reproduce' gets written on the ticket, the bug stays alive, and everyone quietly knows it will be back — at a worse moment.
What we set up
The trace — the recording of a run — captures every input that could have changed the agent's behavior: the event that started it, the exact passages retrieval returned, every tool call's inputs and outputs, the model's settings (its parameters), and the model's response itself. Replay re-runs the same workflow against those captured inputs, so a developer sees precisely what the agent saw that day. In development, real tools are swapped for stand-ins that answer from the recording (stubs), so replaying a run never touches live systems — no refund gets re-issued while someone is debugging. And when a replay diverges from what production actually did, that difference is itself tracked: it means something influenced the run that the trace didn't capture, which is a finding of its own.
How it works, step by step
- Every run is recorded in flight
Starting event, retrieved passages, tool calls and their results, model settings, and the model's response — captured as it happens.
- A bad output gets a trace ID
The ticket points at the exact run, not at a vague description of what someone remembers.
- The run is replayed from the recording
Same inputs, same passages, same tool answers. The developer steps through last Thursday exactly as it unfolded.
- Replays touch nothing real
Stand-in tools answer from the recording, so debugging never triggers a real action twice.
- Differences are a signal
When the replay diverges from what production did, something escaped the recording — and that gap gets fixed too.
What changes for you
Before: agent bugs ended in 'cannot reproduce' and a quiet wait for the problem to strike again. After: debugging becomes ordinary engineering — pull the trace, replay it, watch the wrong turn happen, fix it, replay again to confirm. The scariest phrase in AI operations, 'we don't know why it did that', gets retired. What it won't do: replay can only reproduce what was recorded. A run from before tracing existed, or an input that slipped past capture, stays out of reach — the discipline of recording comes first, and the payoff follows it.