Quality Gates
Hard minimum scores a change must beat before it can go live — enforced by the machinery itself, so a bad version can't ship by accident or forgetfulness.
It's like a driving test before you hand someone the car keys. It doesn't matter how confident they sound — no pass, no keys. Quality gates make every workflow change take the test, automatically, before it's allowed anywhere near production.
The problem, in plain words
There is a quality checklist. It lives on a wiki page (Confluence) that everyone genuinely means to follow. Then one Friday a change ships without the checks — not out of malice; someone was in a hurry and simply forgot — and Monday's inbox is full of complaints about answers that got worse. You do the post-mortem and the conclusion is uncomfortable: a rule that depends on someone remembering it isn't really a rule. It's a wish.
What we set up
On every proposed change, the eval harness (the test bench that runs your saved test cases) produces two things: a quality score and a regression list (the old, already-fixed bugs, re-tested one by one). The gate lives inside CI (the automated pipeline every change passes through on its way to production) — if the score falls below the declared floor, or if any regression case fails, the pipeline itself blocks the change. No human has to remember anything. The gate values are versioned (every adjustment recorded, with who and when), and they can be tightened deliberately over time as the system matures — but never loosened quietly.
How it works, step by step
- A change is proposed
A new prompt, a model swap, a restructured step — anything that could move quality in either direction.
- The tests run automatically
The full eval set runs and produces a quality score plus a pass/fail on every past-bug regression case.
- The gate compares against the floor
Score below the minimum, or any old bug resurfacing, means an automatic no. The rules are written down, not remembered.
- Blocked means blocked
The pipeline refuses to ship it. There is no we-will-fix-it-later path around the gate — the fix comes first.
- Gates tighten as you improve
The floor values are versioned and can be raised on purpose as quality climbs — a ratchet, not a ceiling.
What changes for you
Before, quality regressions were discovered by customers, and 'we forgot to check' was a real failure mode. After, the checking cannot be forgotten, because no human has to remember it — shipping fast and shipping safe stop being a tradeoff for routine changes, and the team's energy goes into raising the floor instead of policing it. What it won't do: it won't measure qualities you never wrote a test for. The gate is exactly as good as the test set behind it — that's why the test set gets its own care.