Batch Pipeline Modernization
The tangle of scheduled scripts that move your data every night gets rebuilt into one tidy, tested system — where every job has an owner, a purpose, and a paper trail.
It's like a kitchen where the recipes live on sticky notes, in three different drawers, in the handwriting of cooks who left years ago. Modernization is copying them into one recipe book: each dish gets a page, an author, and a list of what depends on it — so a new cook can follow along without interviewing the whole kitchen.
The problem, in plain words
Somewhere in your company, a scheduled job (a cron job — a script set to run automatically at a fixed time) has been running every night at 2 a.m. since 2019. Nobody remembers who wrote it. It feeds a report that feeds a spreadsheet that feeds a decision. One night it fails, and the failure message goes to a channel nobody watches. Three days later, a director asks why the numbers look frozen — and the investigation takes a week, because there are hundreds of these jobs spread across three scheduling systems, and the documentation hasn't matched reality since the last reorg.
What we set up
We move the whole tangle onto one orchestrator (Airflow, Dagster, Prefect, or your cloud's equivalent — a conductor that runs every job in the right order and notices when one fails). The data transformations themselves are rewritten in a framework built for the job (dbt or SQLMesh), where each step is versioned like code and ships with tests. Dependencies between jobs become an explicit map the conductor enforces — job B simply doesn't start until job A has finished cleanly. And every job names an owner, so failures go to a person with the context attached, not into the void.
How it works, step by step
- Inventory what actually runs
Every scheduled job in every system gets found and listed — including the ones nobody remembers. This step alone is usually eye-opening.
- Pick one conductor
All jobs move to a single orchestrator. One place to see what runs, what failed, and what's waiting on what.
- Rewrite transformations with tests
Each transformation becomes versioned code with tests, so a change that would break the output is caught before it ships — not discovered on a dashboard.
- Declare the dependencies
The order of operations becomes an explicit map. Downstream jobs wait for upstream ones instead of running on a hopeful timer.
- Route failures to owners
Each job names an owner. When it fails, that person gets the message with the context — not a channel nobody reads.
- Retire the stale jobs
With every consumer visible, jobs that feed nothing can be switched off with confidence instead of superstition.
What changes for you
Before: understanding a pipeline means asking three people and hoping one remembers. After: a new analyst can read what a job does, who owns it, and what depends on it — all in one place. Failures surface in minutes, to the right person, instead of surfacing days later as a frozen dashboard. What it won't do: it won't fix logic that was wrong to begin with. If a job has been calculating something incorrectly for years, modernization makes that job visible and testable — someone still has to decide what correct means.