Skill Distillation

What it is

Skill Distillation is the discipline of converting an AI agent's tacit success into transferable knowledge — the kind a fresh agent with zero context can pick up and execute. Picture a senior engineer who can quietly fix a tricky deploy because she has years of accumulated context, the codebase in her head, the right tools at her fingertips, and the stubbornness to keep trying until it works. Now picture handing that same task to someone on their first day. They have the same instructions; they fail. The difference is everything that was implicit in the senior engineer's head. Skill distillation is the discipline of making that implicit knowledge explicit, with a test that proves the artifact actually works.

Why it matters

Every organization that uses AI is going to face this problem within a year. Your most productive AI use today is happening in someone's personal Cursor or Claude Code, where they've accumulated context, MCP servers, project conventions, and a history of working fixes. That setup is irreproducible: if they leave, if they go on vacation, if a teammate tries to do the same task, the magic doesn't transfer. Skill distillation is how a team's actual craft — the thing that makes its agents succeed — becomes durable. Without it, every onboarding starts over and every offboarding loses something.

How it actually works — the mentor–apprentice loop

Two roles, not two models. The mentor is the educated agent — it carries the context and it knows what success looks like. The apprentice is a fresh, isolated agent with no prior memory. The mentor authors the skill document, hands it to the apprentice, and the apprentice attempts the task with only that document. The mentor grades the output against the expected result. If the apprentice missed, the mentor diagnoses what was tacit — the unspoken step, the missing example, the assumption about a tool — updates the artifact, and re-runs the apprentice. The loop stops when the apprentice succeeds without any mentor intervention beyond the artifact itself. One case is not enough: as new cases arrive they become regression tests, and any future revision must clear the full suite before it counts as an improvement.

• Mentor drafts the skill document from tacit context
• Fresh apprentice runs the task with only the document
• Mentor grades against the expected outcome
• Mentor revises the document; apprentice re-runs
• Each accepted case enters a growing regression suite
• Convergence is the apprentice succeeding without help across the suite

What makes a skill document hold up

Naming tacit failure modes the mentor would normally just feel ('this tool 5xx's about 1 in 20 — retry with this backoff, do not escalate'); enumerating tool surfaces with realistic examples instead of abstract specs; including recovery paths for the failures that actually happen, not the textbook ones; baking verification steps in so the apprentice can confirm success on its own; and proving completeness with a clean apprentice run rather than a mentor review. The artifact is good when it can be handed to a teammate, a new vendor, or a different model and still work.

What it works with

Sits beside Closed-Loop Knowledge. The two are complementary loops: closed-loop knowledge improves the system from production failures; skill distillation improves the artifacts that an apprentice agent reads. The regression suite mechanism is borrowed from Workflow Evals — the discipline is the same, applied to skill documents instead of model outputs. The apprentice agent runs through the AI Platform like any other; the mentor uses its own context plus the Source Graph it has access to. Synthetic Personality is the layer the apprentice does not touch — skill documents describe what to do, not who to be.

Open questions we are studying

When does the mentor's grade diverge from a downstream user's grade — does converging the loop guarantee real-world success, or only success against the mentor's mental model? How big is the gap between 'the apprentice ran this case' and 'the apprentice can run a sibling case the mentor did not anticipate'? How do we detect when a document has overfit to the regression suite rather than to the underlying skill? What is the right substrate for the documents — Markdown, a structured schema, an executable runbook, a small program — given that the apprentice will read it with a model whose preferences shift over time?

Prior art and adjacent work

Builds on knowledge and context distillation work in ML. Engages with skill libraries (Voyager — Wang et al. 2023, Eureka — Ma et al. 2023) and the broader research on agents that author their own skill repertoires. Adjacent to apprenticeship learning and inverse reinforcement learning. Borrows golden-test and TDD discipline from software engineering for the regression suite.