Source Contracts
A written agreement for every data source — what shape the data arrives in, how fresh it must be, and who gets called when it breaks — checked automatically at the door.
Like a delivery agreement with a supplier: the boxes arrive at an agreed time, packed an agreed way, and if a shipment is wrong you know exactly who to call — instead of unpacking mystery boxes and guessing who sent them. A source contract is that agreement, for data, enforced by a machine.
The problem, in plain words
A pipeline breaks overnight. In the morning your dashboard is blank, and the detective work begins: which of the forty sources changed? The CRM export, the payments feed, the partner's spreadsheet? You dig through logs, find a renamed field, then spend the afternoon figuring out which upstream team did it and whether they even know. By the time it's fixed, two days of records have been lost or mangled — and the downstream teams have quietly gone back to not trusting the data.
What we set up
Every source gets a contract, written in code: exactly what the data looks like (a typed schema — every field named, with its type), how fresh it must be and how that's measured, how many errors are tolerable before it counts as broken (an error budget), and what happens when the contract is violated — retry, set the bad records aside, or stop the line. Validation runs the moment data arrives. The contract is versioned like code, and downstream teams subscribe to it — so a planned change is announced to everyone affected, instead of discovered by them.
How it works, step by step
- Write the contract per source
Shape, freshness, error budget, and breakage policy — agreed with the team that owns the source, and stored as code.
- Check every delivery at the door
Incoming data is validated against the contract on arrival, not after it has spread downstream.
- Handle violations by the agreed policy
Depending on the contract: retry, divert bad records to a dead-letter table (a lost-and-found), or stop and refuse the load.
- Page the owner automatically
When a contract breaks, the source's named owner is notified with the failing records attached — no forensics needed to find the culprit.
- Version changes and notify subscribers
A source that needs to change its shape publishes a new contract version; every subscribed consumer hears about it before it happens.
What changes for you
Before: a broken pipeline is a whodunit that eats your data team's week. After: it's a notification to a named owner, with the evidence attached and the bad records safely set aside. Downstream dashboards and agents learn they can trust what arrives — and freshness telemetry lets retrieval skip a source that has gone quietly stale. What it won't do: it won't stop sources from ever breaking. Systems change and mistakes happen; the contract makes breakage visible, contained, and attributable — not impossible.