Eval Case Capture
When a conversation is unusually good or unusually bad, one flag saves it into your test set — so the great answers become the standard and the failures can never quietly return.
Think of a coach clipping game film: the beautiful play gets saved so the team can repeat it, the fumble gets saved so it never happens again. Eval case capture is the clip button for your AI's conversations — the interesting ones stop scrolling away and start making the system better.
The problem, in plain words
Every week your system produces a few conversations worth learning from: an answer so good you would want every future version to match it, and a failure so instructive you never want to see it again. And every week, both scroll away. Then someone changes a prompt or a model gets updated, and the failure you fixed in March quietly comes back in August — because nothing was ever written down that would catch it. The team keeps saying we should write evals someday, and someday stays someday.
What we set up
We add a capture step to work your reviewers already do. While triaging conversations, they flag the interesting ones — one action, in the flow, not a separate chore. Each flagged case goes through a quick labeling step: gold (an example to preserve — future versions must do at least this well), regression (a failure to prevent — future versions must never do this again), or ambiguous (worth keeping, needs discussion). Labeled cases land in the eval set (your bank of test cases the system is measured against) with attribution, so you know who captured what and why. Then the test harness re-runs on the enlarged set — every capture immediately becomes part of the safety net, and its impact on the scores is measurable.
How it works, step by step
- A conversation stands out
Unusually good, badly failed, or strange in an instructive way — a reviewer notices it during normal triage.
- The reviewer flags it
One action, inside the triage flow. No separate tool, no write-it-up-later.
- It gets a quick label
Gold to preserve, regression to prevent, or ambiguous for discussion — a short labeling step, not a research project.
- It joins the test set
With attribution: who captured it and why. The test set grows from your real traffic, not from invented scenarios.
- The harness re-runs
Every future change is now tested against this case too. The March failure cannot quietly return in August.
What changes for you
Before: lessons live in memory, the test set was written once and froze, and old bugs reappear whenever something changes upstream. After: the test set grows a little every week from real conversations, gold cases define the bar and regression cases guard the floor, and the write-evals-later debt stops accumulating. What it won't do: it won't flag cases by itself — capture depends on reviewers noticing, so conversations nobody reviews are conversations nobody captures.