Drift Alerts
A regular automated check that notices when your AI starts behaving differently than it used to — and tells you before your customers do.
Think of a smoke detector for your AI system. It does not put out fires, but it goes off early — when the model behind your assistant gets quietly updated, when your documents age, or when a small prompt change shifts the answers — so you find out from an alert, not from an angry customer.
The problem, in plain words
Here is the uncomfortable truth about AI systems: they change even when you change nothing. The company behind the model can update it under the same name, and answers that used to be crisp get longer, or more cautious, or just different. Your help docs slowly go stale. Someone copy-pastes a prompt tweak into one workflow and not another. None of this shows up as an error — nothing crashes, no red light blinks. The system just drifts, a little each week, until one day a customer points out that the assistant has been confidently wrong for a month, and nobody can say when it started.
What we set up
We set up a scheduled comparison between how your system behaves in production today and how it behaved when you last tested and approved it. The reference point is your eval set (a collection of test questions with answers you have already checked and signed off on). On a schedule, the system replays a sample of work and compares simple, honest measurements: how often the AI refuses to answer, how long its answers run, how the tone reads, how often it cites its sources, and how it scores on held-out test questions. When a measurement moves past the threshold you chose — and you can tune that threshold for each workflow — an alert fires, with a candidate cause attached: a model update, a document change, a prompt edit.
How it works, step by step
- A baseline is recorded
When your system is tested and approved, its behavior — refusal rate, answer length, tone, citation rate, test scores — is saved as the reference point.
- The check runs on a schedule
At regular intervals, the system replays a sample of work and measures the same things again. Nobody has to remember to do it.
- Today is compared to the baseline
Small wobbles are normal and get ignored. Real shifts — past the threshold you set for that workflow — are what count.
- An alert fires, with a suspect
The alert does not just say something changed. It names a likely cause: the model was updated, a document aged out, a prompt was edited.
- You tune the sensitivity
Each workflow gets its own thresholds, so a chatty marketing assistant and a strict billing agent are judged by their own standards.
What changes for you
Before: you learn about drift when a customer complains, then spend days working out when the change happened and what caused it. After: you get an alert soon after the shift, with a likely cause already attached, and you can react while the problem is still small. Drift stops being a mystery and becomes a maintenance item, like a low-battery chirp. What it won't do: it won't fix the drift for you — it detects and explains, and a person still decides what to change.