Scout Evals
We take 60 days of your production AI traces, build a real eval set, and hand it back with a working harness — so you can finally measure whether your AI is getting better or quietly worse.
What you get
After 1-2 weeks: a curated eval set built from your last 60 days of production traces — gold examples, regression cases, calibrated against a small human-rated set — plus a working harness that runs candidates against it and produces a diff report. You can keep building on it; we can keep building on it with you; it's not locked to us.
What makes it different
Most teams know they should have evals; almost none do, because building the first one is hard, time-consuming, and feels unrewarded. Scout Evals does the hard first pass so the team starts from something real instead of staring at a blank Inspect AI config.
What it can do for you
Reveal whether your AI is actually getting better or quietly drifting (most teams discover the latter). Make prompt and model changes reviewable. Give you a number for 'is this change safe to ship.' Catch regressions before they hit production.
- 1-2 week engagement, fixed scope
- Eval set: gold + regression + calibration
- Working harness on Inspect AI, OpenAI Evals, or Promptfoo
- Promotion-gate template you can wire into your CI
- Drift-detection baseline to compare against weekly
- Calibration audit on LLM-as-judge
- All artifacts yours; no platform lock-in
- Optional handoff training for your team
Who it's for
Teams shipping AI to production who can't currently answer 'is this change better or worse.' Teams who tried to build evals themselves and stalled. Teams who suspect their AI is degrading but lack the data to prove it.
How we ship it
1-2 weeks. Week 1: trace ingestion, triage, eval-set construction, calibration. Week 2 (if needed): harness setup, CI integration, drift-baseline run.
Questions people actually ask
What exactly do you need from us to build the eval set?
Your last 60 days of production AI traces — the real conversations or runs your system produced. That's the raw material; without production traces there's nothing honest to build from, and we'll tell you so rather than fabricate a synthetic set that measures nothing.
Our traces contain customer data. How is that handled?
Access is scoped and agreed before ingestion, and the eval set is built for your environment, not ours: gold examples, regression cases, and a calibration set rated by humans. Every artifact — the eval set, the harness, the baselines — is yours, with no platform lock-in.
How long does it take and what do we end up running?
1 to 2 weeks. Week 1 covers trace ingestion, triage, eval-set construction, and calibration; week 2, if needed, sets up the harness — on Inspect AI, OpenAI Evals, or Promptfoo — plus CI integration and a drift-detection baseline. You end up with a working system that answers whether a change is safe to ship.
Can our team maintain it without you afterward?
Yes — that's the design. The harness runs on open tooling, the promotion-gate template wires into your own CI, and optional handoff training gets your team comfortable extending it. We can keep building with you if you want, but nothing about the setup requires it.
Frequently paired with
Related products
A one-week engagement that maps your AI-readiness — what's in your data substrate, what's missing, what AI capabilities you could deploy today, and what would need to be built first.
A near-free developer tool: upload a persona charter, paste a model API key, run it against our probe library, see exactly where your persona holds and where it drifts before you ship.