Lineage Mapping
A family tree for your data — where every table and column comes from and everything it feeds — so 'if I change this, what breaks?' takes a query instead of a week.
Like the ingredient trail on food packaging: from this farm, through this mill, into this loaf. Lineage is that trail for your data — every number on a dashboard can be traced back through each transformation to the source it came from, and forward to everything that would notice if it changed.
The problem, in plain words
Someone wants to rename a column, or retire an old table, and asks the fatal question: 'is anyone still using this?' The honest answer is that nobody knows. So either the change is made bravely — and three weeks later, a report in another department quietly breaks — or the change isn't made at all, and the warehouse keeps accumulating tables nobody dares touch. Meanwhile an auditor asks where a regulatory figure comes from, and the answer takes a week of archaeology.
What we set up
We make every tool in the pipeline report what it reads and writes. Emitters using an open standard (OpenLineage) attach to each engine — the transformation tool (dbt), the processing engines (Spark, Flink), the orchestrator (Airflow) — and every run reports which columns it read and which it wrote. Those reports feed a lineage backend (Marquez, DataHub, Atlan, or OpenMetadata) that assembles the full family tree, down to individual columns. The graph is queryable, and it connects to the data catalog's permissions — so 'who can see anything derived from this sensitive column?' is one question, not a research project.
How it works, step by step
- Every engine reports its moves
Each pipeline run logs what it read and what it wrote, in a shared open format, automatically — nobody maintains a diagram by hand.
- The reports build the tree
The lineage backend stitches the run reports into one graph: source to table to column to dashboard.
- Ask it before you change anything
An impact-analysis query lists everything downstream of a column — every table, dashboard, and consumer that would feel the change.
- Trace any number to its source
From a dashboard figure back through each transformation to the raw records. Audits follow the trail instead of interviewing people.
- Cross it with permissions
Because the graph connects to the catalog, sensitive data can be followed through its derivatives — who can see what came from what.
What changes for you
Before: schema changes are made bravely or not at all, and audits are archaeology. After: any change starts with a query that lists exactly what is affected, retiring a table starts with the list of its real consumers, and 'where does this number come from?' has a visual answer. What it won't do: it can only trace what flows through instrumented tools. The spreadsheet someone maintains by hand on their laptop is invisible to the graph — until its process moves into the pipeline.