Root-Cause Analysis
Figuring out which part of the system actually caused a batch of bad conversations — the search, the reasoning, the tools, the documents, or the rules — so the fix goes where the problem lives.
If your tire keeps going flat, patching it again is not the fix — finding the nail in your driveway is. Root-cause analysis is that, for AI conversations: instead of rewriting the prompt for the third time, it works out which part of the system actually failed, and sends the fix there.
The problem, in plain words
A cluster of bad conversations shows up and the team does what teams do: tweaks the prompt. Sometimes that works. Often the prompt was never the problem — the search pulled up the wrong documents, or the documents themselves were out of date, or the AI reasoned fine but called the wrong tool, or a safety rule blocked an answer it should have allowed. Treat every failure as a prompt problem and you will ship the same prompt fix three times while the real cause sits untouched — and the cluster comes right back.
What we set up
We take a sample of threads from a bad cluster and replay them with the full trace (the complete record of what the system did at each step: what it searched, what it found, how it reasoned, what it called). With the trace open, each failure gets sorted into one of five causes: retrieval — the search returned the wrong documents; reasoning — right documents, wrong conclusion; tool — right conclusion, wrong action taken; knowledge — the source document itself was wrong or stale; or policy — the answer was right but a rule wrongly blocked it. The cluster is then routed to the team that owns that cause, and closure is tracked per category, so you can see over time which part of your system generates the most trouble.
How it works, step by step
- A bad cluster is selected
A group of related failures, usually straight from bad-thread clustering, becomes the case to solve.
- A sample is replayed with traces
For each sampled thread, the full step-by-step record is opened: what was searched, found, reasoned, and done.
- Each failure is diagnosed
It gets one of five causes: retrieval, reasoning, tool, knowledge, or policy — each pointing at a different part of the system.
- The fix routes to its owner
A retrieval problem goes to the search team, a stale document to its author, a wrong refusal to whoever owns the rules.
- Closure is tracked by category
Fixed clusters are watched for recurrence, and cause categories are counted over time — showing where the system is weakest.
What changes for you
Before: fixes chase symptoms — prompts get rewritten on instinct, some clusters improve for reasons nobody can explain, and others return in a month. After: each cluster carries a diagnosis, the fix lands in the part of the system that actually failed, and recurrence drops because the nail came out of the driveway. What it won't do: it won't diagnose a single one-off complaint — it needs a cluster of related failures and their traces to be worth the effort.