Generated-Code Tests
When the AI writes code that runs inside a workflow, that code gets its own tests — so a change can't quietly break a piece nobody thought to check.
If a student builds their own measuring tool for a science project, a good teacher checks the tool, not just the results. Same idea here: when an agent writes code — small handlers, converters, checkers — that code gets tested directly, not just the instructions that produced it.
The problem, in plain words
You tweak a prompt. The prompt tests pass, everything is green, the change ships. A week later, a small data converter the agent generated months ago starts mangling dates — and nobody notices for days, because nobody's tests covered it. Everyone was testing the model's answers; no one was testing the code the model wrote. That gap is where 'the prompt is fine' and 'the workflow is broken' can both be true at the same time.
What we set up
Every piece of code the agent generates — handlers (small programs that react to events), transformers (code that converts data from one shape to another), validators (code that checks data is well-formed) — lands together with test cases built from real inputs your workflow has actually seen, not toy examples. The eval harness (the test bench that scores every change) runs those code tests alongside the prompt tests, on every variant. Where coverage is thin, the gap is surfaced as a request for new test cases instead of staying invisible, and failures from production are added to the regression set (the collection of past bugs every future change must clear).
How it works, step by step
- The agent generates code
A handler, a converter, a checker — a piece of code that will run inside the workflow, every day, on real data.
- Tests arrive with it
Built from real inputs the workflow has seen, so the tests exercise the cases that actually happen.
- Every change runs the tests
Change a prompt, swap a model, restructure a step — the code tests run too, every time, not just when someone remembers.
- Gaps get flagged
Parts of the code no test touches show up as work to do now, instead of surprises later.
- Real failures become permanent tests
When something breaks in production, that case joins the test set — so it can never break silently again.
What changes for you
Before, generated code was the workflow's blind spot: the piece that failed silently while all the visible checks stayed green. After, it breaks loudly, in testing, before customers ever see it — and a 'safe' prompt change can no longer ship a broken handler as a side effect. What it won't do: it won't guarantee the code is elegant or fast. The tests check that it does what the workflow needs — which is the part that hurts when it's wrong.