Data Quality Gates
Checkpoints along your data's journey that stop bad records before they spread — so problems get caught at the door, not on an executive's dashboard.
Think of quality control on a factory line. You inspect parts as they arrive, again after assembly, and once more before shipping — because a defect caught early is a small fix, and the same defect found by a customer is a very bad day. Data quality gates are those inspection stations, for data.
The problem, in plain words
The quarterly numbers look off. An executive notices before the data team does. Now the pressure is on: three people drop what they were doing and dig backwards through the pipeline — dashboard to table to job to source — until, a week later, they find it. A source system started sending empty values in one field a month ago, and every calculation downstream quietly absorbed them. The bug took a minute to fix. Finding it took a week. And the numbers had been wrong in front of the leadership team the whole time.
What we set up
We install checkpoints at three places in the pipeline, using established validation tools (Great Expectations, Soda, dbt tests, or Monte Carlo). At the door — when data arrives — gates check the shape and the share of missing values. Mid-journey — during transformation — gates check business rules: totals that must add up, dates that can't be in the future, amounts that can't be negative. At the exit — before data is published to dashboards and agents — gates check freshness and row counts, catching the day a feed silently delivers half its usual volume. When a gate fails, the flow stops there, and the owner of the failing source is notified with the bad records attached.
How it works, step by step
- Check the shape at the door
Arriving data is tested for the right structure and a normal rate of missing values. Wrong shape? It stops here.
- Check the sense mid-journey
Business rules run during transformation: sums that must reconcile, values within plausible ranges, dates that make sense.
- Check the pulse before publishing
Freshness and volume are verified last — data that is stale or suspiciously thin never reaches the dashboards.
- Stop the line on failure
A failed gate blocks the bad batch from moving downstream. Yesterday's good data stays in place; the flood of wrong numbers never happens.
- Page the owner with the evidence
The team that owns the failing source gets the alert, with the exact records that failed — so the fix starts immediately, with context.
What changes for you
Before: bad data is discovered downstream, by the wrong people, at the worst moment — and the cleanup is forensic. After: it's caught at a gate, attributed to a source, and fixed by the team that owns it — often before anyone downstream noticed. Your data team stops taking the blame for other teams' upstream issues. What it won't do: it can only catch what a rule tests for. A value that is wrong but plausible — a real-looking price that is simply incorrect — will pass through, unless someone writes the rule that would catch it.