Skill Distillation
How an educated AI agent — the one with the codebase, the tools, and the tacit context that lets it succeed — distills its competence into transferable skill documents a virgin agent can run from scratch.
Production AI is not a prompt. It is a system of context, tools, permissions, traces, evals, and feedback loops — including a deliberate path for converting tacit competence into transferable skill documents that hold up across cases.
What it is
Skill Distillation is the discipline of converting an AI agent's tacit success into transferable knowledge — the kind a fresh agent with zero context can pick up and execute. Picture a senior engineer who can quietly fix a tricky deploy because she has years of accumulated context, the codebase in her head, the right tools at her fingertips, and the stubbornness to keep trying until it works. Now picture handing that same task to someone on their first day. They have the same instructions; they fail. The difference is everything that was implicit in the senior engineer's head. Skill distillation is the discipline of making that implicit knowledge explicit, with a test that proves the artifact actually works.
Why it matters
Every organization that uses AI is going to face this problem within a year. Your most productive AI use today is happening in someone's personal Cursor or Claude Code, where they've accumulated context, MCP servers, project conventions, and a history of working fixes. That setup is irreproducible: if they leave, if they go on vacation, if a teammate tries to do the same task, the magic doesn't transfer. Skill distillation is how a team's actual craft — the thing that makes its agents succeed — becomes durable. Without it, every onboarding starts over and every offboarding loses something.
How it actually works — the mentor–apprentice loop
Two roles, not two models. The mentor is the educated agent — it carries the context and it knows what success looks like. The apprentice is a fresh, isolated agent with no prior memory. The mentor authors the skill document, hands it to the apprentice, and the apprentice attempts the task with only that document. The mentor grades the output against the expected result. If the apprentice missed, the mentor diagnoses what was tacit — the unspoken step, the missing example, the assumption about a tool — updates the artifact, and re-runs the apprentice. The loop stops when the apprentice succeeds without any mentor intervention beyond the artifact itself. One case is not enough: as new cases arrive they become regression tests, and any future revision must clear the full suite before it counts as an improvement.
- Mentor drafts the skill document from tacit context
- Fresh apprentice runs the task with only the document
- Mentor grades against the expected outcome
- Mentor revises the document; apprentice re-runs
- Each accepted case enters a growing regression suite
- Convergence is the apprentice succeeding without help across the suite
What makes a skill document hold up
Naming tacit failure modes the mentor would normally just feel ('this tool 5xx's about 1 in 20 — retry with this backoff, do not escalate'); enumerating tool surfaces with realistic examples instead of abstract specs; including recovery paths for the failures that actually happen, not the textbook ones; baking verification steps in so the apprentice can confirm success on its own; and proving completeness with a clean apprentice run rather than a mentor review. The artifact is good when it can be handed to a teammate, a new vendor, or a different model and still work.
What it works with
Sits beside Closed-Loop Knowledge. The two are complementary loops: closed-loop knowledge improves the system from production failures; skill distillation improves the artifacts that an apprentice agent reads. The regression suite mechanism is borrowed from Workflow Evals — the discipline is the same, applied to skill documents instead of model outputs. The apprentice agent runs through the AI Platform like any other; the mentor uses its own context plus the Source Graph it has access to. Synthetic Personality is the layer the apprentice does not touch — skill documents describe what to do, not who to be.
Open questions we are studying
When does the mentor's grade diverge from a downstream user's grade — does converging the loop guarantee real-world success, or only success against the mentor's mental model? How big is the gap between 'the apprentice ran this case' and 'the apprentice can run a sibling case the mentor did not anticipate'? How do we detect when a document has overfit to the regression suite rather than to the underlying skill? What is the right substrate for the documents — Markdown, a structured schema, an executable runbook, a small program — given that the apprentice will read it with a model whose preferences shift over time?
Prior art and adjacent work
Builds on knowledge and context distillation work in ML. Engages with skill libraries (Voyager — Wang et al. 2023, Eureka — Ma et al. 2023) and the broader research on agents that author their own skill repertoires. Adjacent to apprenticeship learning and inverse reinforcement learning. Borrows golden-test and TDD discipline from software engineering for the regression suite.
Related resources
How an AI system gets durably better at its job — not by being smarter, but by routing every production failure into either a knowledge update, an eval case, a workflow patch, or a documented exception with a named owner.
The test suite your AI workflows have to pass before any change reaches users — measuring quality, latency, cost, and safety on real production data instead of vibes.
Before an AI agent can be useful to anyone, it has to be something — a coherent identity that holds up across users, sessions, and adversarial pressure. This is the research track that defines what that means and how to keep it stable.