Workflow Mutation
Instead of arguing about how to improve an AI workflow, we generate lots of variations and let a test bench score them — only proven winners get shipped.
Think of a test kitchen. Before a restaurant changes its menu, the chef tries ten versions of a dish side by side and keeps the one that actually tastes better. Workflow mutation does that for AI workflows: many candidate versions, all scored on the same tests, before any of them gets near your customers.
The problem, in plain words
Someone on your team is sure a shorter prompt would work better. Someone else wants to try the new model everyone's talking about. A third person thinks the workflow fetches too many documents. Today, the loudest opinion wins, the change ships, and production traffic becomes the taste test — your real customers, judging a change nobody measured. If quality dips, you find out slowly, from complaints, and untangling which change caused it is guesswork.
What we set up
We define mutation operators — small, systematic recipe-changers — for each part of a workflow: rewriting the prompt (the agent's instructions), swapping the model, fetching more or fewer documents (retrieval depth), restructuring the steps, and tightening or loosening the tool budget (how many tool calls the agent is allowed). The eval harness (the test bench that runs every candidate against a saved set of test cases) tries the combinations — or a guided search when there are too many to try them all — and reports the candidates that are better on at least one measure without being worse on the others (Pareto-dominant, in the jargon). Nothing is promoted without a human saying yes.
How it works, step by step
- Pick what to vary
Prompts, models, retrieval depth, step structure, tool budgets — each one is a dial the search is allowed to turn.
- Generate the candidates
The system produces many variant workflows automatically, instead of one hand-made change based on a hunch.
- Score them all the same way
Every candidate runs against the same saved test cases. Same questions, same grading — a fair race.
- Keep the genuine winners
Candidates that improve quality, cost, or speed without losing ground elsewhere rise to the top of the report.
- A person approves the promotion
The winning variant is a recommendation with a scorecard, not an automatic deploy. A human clicks yes before it replaces the current version.
What changes for you
Before, workflow changes shipped on opinion and got judged by production traffic. After, changes arrive with a scorecard, and the ones that don't measure up stay in the variant table instead of in front of customers. Improvement stops being an argument and becomes a process that runs on evidence. What it won't do: it won't invent your definition of better. The test cases decide what counts as an improvement — so the search is exactly as good as the tests it is scored on.