Scout Evals

What you get

After 1-2 weeks: a curated eval set built from your last 60 days of production traces — gold examples, regression cases, calibrated against a small human-rated set — plus a working harness that runs candidates against it and produces a diff report. You can keep building on it; we can keep building on it with you; it's not locked to us.

What makes it different

Most teams know they should have evals; almost none do, because building the first one is hard, time-consuming, and feels unrewarded. Scout Evals does the hard first pass so the team starts from something real instead of staring at a blank Inspect AI config.

What it can do for you

Reveal whether your AI is actually getting better or quietly drifting (most teams discover the latter). Make prompt and model changes reviewable. Give you a number for 'is this change safe to ship.' Catch regressions before they hit production.

• 1-2 week engagement, fixed scope
• Eval set: gold + regression + calibration
• Working harness on Inspect AI, OpenAI Evals, or Promptfoo
• Promotion-gate template you can wire into your CI
• Drift-detection baseline to compare against weekly
• Calibration audit on LLM-as-judge
• All artifacts yours; no platform lock-in
• Optional handoff training for your team

Who it's for

Teams shipping AI to production who can't currently answer 'is this change better or worse.' Teams who tried to build evals themselves and stalled. Teams who suspect their AI is degrading but lack the data to prove it.

How we ship it

1-2 weeks. Week 1: trace ingestion, triage, eval-set construction, calibration. Week 2 (if needed): harness setup, CI integration, drift-baseline run.