An improvement that skips review is drift

A support team ran a triage agent on a public issue tracker. Its procedure told it to read each new issue, label it, and search two places for an existing report: the repo itself and the sibling repo that ships the SDK. An improvement agent read the triage agent's failed runs, saw a pattern in how vague bug reports were labelled, and rewrote the procedure file from those failures alone. The new file was shorter and read better. It also lacked the sibling-repo search, because none of the failures had ever reached that step. The change merged on a Monday after one approving click and no test. For nine days, every issue that duplicated one in the sibling repo was created a second time, and two contributors filed separate pull requests for the same bug. Maintainers could not tell which report was canonical. Nobody could say which failures the rewrite answered, what the old file had contained, or which version to restore, so nobody could decide what to revert. Your factory has files like this one, and every one of them changes agent behaviour the moment it is saved. Decide now: what must an agent's proposed edit to one of them contain before anyone is allowed to merge it?

The deliverable is factory/improve/skill-change.md — one proposal format, one review gate, one revert line and a weekly churn budget for changes to the files that govern the agents. It serves the Prove stage of this course's loop (Slice → Gate → Run → Wrap → Prove): Prove decides whether a line earned autonomy, and this file stops the evidence behind that decision from being spent on an unreviewed rewrite of the line itself.

What is actually going on

A skill is a text file. The Agent Skills specification requires only name and description in the frontmatter, offers a free-form metadata map for anything else, and requires no version field (Agent Skills specification, first-party, observed 2026-10-29). A version exists only if you write one into metadata, and the history of the file is whatever your version control keeps.

Three properties follow. First, behaviour changes the moment the file changes: unless you build a gate, nothing stands between the edit and the next run that loads the skill. Second, the file carries instructions and no access. Warp's docs describe skills as providing "instructions, not access to tools or repositories" (Warp docs, first-party, observed 2026-10-29). A bad edit cannot grant a permission, but it can remove a step, and a removed step raises no error. Nothing fails; a check just stops happening. Third, an improvement agent that reads only failures sees only failing paths. A passing run tells it nothing about which lines mattered, so a rewrite that omits a line looks the same as a rewrite that kept it, until a task needs the line. That is how the sibling-repo search disappeared.

Warp's public design shows the shape of an answer. A factory definition is a versioned tree of YAML and Markdown, and a GitHub-backed one "goes through pull request review like any other code" (Warp docs, first-party, observed 2026-10-29). Self-improvement turns a scorer's repeated failures into follow-up pull requests you review. The default trigger is 25 distinct unreviewed failures or an unreviewed failure that is seven days old, failedRunThreshold accepts 1 to 50, and reviewerType defaults to admins with one eligible reviewer requested at random unless you choose none (Warp docs, first-party, observed 2026-10-29). Review is a configured stage of the loop, which also means it is a stage someone can configure away.

Follow the two exits that lead back to the old file, one at review and one after deploy.

The diagram showed that merging is not the end of the change: a watch window sits between deploy and trust, and revert has the same standing as accept. The two exits that matter behave differently. Rejection costs nothing, because the file never changed and the failures return to the cluster as evidence. Revert costs a period of bad behaviour, and it works only if the command was written and tested before the merge. A change with no written exit has one way out, which is a person reading history under pressure.

The rule and its condition

Change a procedure file only through a proposal that names four things: the failures it fixes, each linked to a run record; the exact diff; the holdout task that would catch a regression; and the revert command. Use the reviewed proposal when the failures are attributable, meaning each linked run failed the scorer and the cause traces to a specific step. Otherwise change nothing, and file the observation as evidence for the next cluster.

The "otherwise" branch is not a refusal to improve. A hunch recorded as evidence counts toward the next cluster, and a cluster that reaches your threshold becomes a proposal. What the branch refuses is an edit with no failure behind it. A public demo of exactly this loop shows what the guardrails look like in practice (screen read from a vendor's demo repository, observed 2026-10-29). The improvement skill could not touch the classification buckets, the comment marker or the feedback footer; the learned-guidelines list had a hard ceiling, with the instruction to consolidate or prune the weakest-evidenced items once it filled; every accepted change bumped the skill's version in the frontmatter, in the file's own version section and in the marker the agent posts into each issue, so any comment could be traced back to the guideline set that produced it; and when the feedback was weak or conflicting the run was required to change nothing and report that no change was warranted. The version trail is the part teams skip. Without it, a bad outcome cannot be tied to the guidelines that were in force when it happened.

The author of the proposal never holds the merge key either (Lesson 04, The Author Never Holds the Merge Key), so the agent drafts and a human decides.

One more clause keeps the rule honest: every deleted line needs its own justification, either a failure row or a written reason. Zero deletions is the default for an evidence-driven fix, because evidence about failures says nothing about the lines that never failed.

Five branches decide what happens to an edit, so here is the tree.

Worked case

You own the triage agent. Its skill is skills/triage-issue/SKILL.md, version "3", owner noam. This week the improvement agent clustered six failed runs and handed you a full rewrite of the file. All six re-run clean on the rewrite. You have one reviewer slot of ten minutes, and the file that would govern this change does not exist yet. That file is the deliverable, factory/improve/skill-change.md — one proposal format, one review gate, one revert line and a weekly churn budget for changes to the files that govern the agents, and this change is its first customer.

There are three options.

• A. Merge the rewrite. It fixes all six failures and reads better than version 3.
• B. Send it back and require a proposal in a fixed format, with the diff cut to the lines the failures justify.
• C. Reject it and wait for a bigger cluster.

Decide now: A, B or C.

The answer is B. Work through it in order.

1. The failures share one defect. In each, step 3 applied ready to a bug report with no reproduction steps, version or environment. Six failures justify one new step and nothing else in the file.
2. The rewrite is not a diff. A reviewer who reads only the new file sees a coherent procedure, and a coherent procedure with a missing step looks complete. The loss shows only against version 3, so the proposal carries the exact diff, including the version bump inside metadata.
3. Narrowing the diff keeps unevidenced lines. Version 3 step 4 searches this repo and acme-sibling. No failure touched it, so the proposal may not remove or reword it.
4. The holdout tests the neighbour as well as the fix. H1 covers the new step. H2 covers the sibling search that the six failures never reached: it passes on version 3 and must still pass on version 4. Its pass criterion is exact, zero new issues and one comment (Lesson 05, Holdouts Need Exact Figures). H2 is the task that would have caught the nine days.
5. The revert line is written and dry-run before merge, because after merge the person who needs it may be reading an incident channel.

The proposal opens with frontmatter. The version bump is stated twice on purpose, once here and once in the diff, so a mismatch is visible at a glance.

---proposal: 2026-w44-triage-needs-reproskill: triage-issueskill_path: skills/triage-issue/SKILL.mdfrom_version: "3"to_version: "4"owner: noamauthor: improvement-agentstatus: proposed---

Then the evidence, one row per failure, each pointing at a run record a reviewer can open.

Then the exact diff. It adds one step, bumps one version, and deletes nothing else.

--- a/skills/triage-issue/SKILL.md+++ b/skills/triage-issue/SKILL.md@@ -4,3 +4,3 @@ metadata:-  version: "3"+  version: "4"   owner: noam@@ -12,3 +12,6 @@ 2. If the issue is spam, close it and stop. 3. Classify the issue as `bug`, `feature` or `question`.+3a. If it is a `bug`, check the body for reproduction steps, the+    affected version and the environment. If any is missing, apply+    `needs-repro`, comment asking for them, and stop. Never apply `ready`. 4. Search this repo and `acme-sibling` for open issues with the same symptom.

Then the holdout. H1 is expected to fail on version 3 and pass on version 4. H2 and H3 must pass on both.

And the revert line, written last and run first on a branch.

git revert --no-edit <merge-sha>   # restores skills/triage-issue/SKILL.md to version "3"

That proposal is one instance. The file that makes every later proposal look the same is skill-change.md: the format, a ten-minute review gate, the revert line and the budget. Build it exactly like this.

Option A's cost is the nine days from the opening. With no diff to read, no holdout and no record of which failures the rewrite answered, the first sign of the missing search is a duplicate issue in the tracker. Option C's cost is smaller and real: bug reports keep landing as ready without steps for another week, though the evidence stays in the cluster.

The budget does the work the gate cannot, which is to limit how many changes reach you at once.

Illustrative composite. Modeled, not measured: a factory with 4 agents, each owning 3 skills. If the improvement agent files one proposal per skill per week and most are approved, 12 behaviour changes land in a week, and a Friday dip in the pass rate has 12 suspects and 2 hours of review behind it at 10 minutes each. Under the budget, 2 land, the review is 20 minutes, and with one skill per agent at a time the dip has at most one suspect per agent.

Faded case

Skip to the open case if this is familiar.

Same shape, different skill, and one decision left to you. skills/summarize-ci-failure/SKILL.md is at version "7" and runs a few times a day, so the team set its threshold at two real failures. The improvement agent filed a proposal with three rows of evidence.

The diff has two hunks. Hunk (a) tells the agent to find the first ERROR line in the full log and quote ten lines around it. Hunk (b) adds a rule that every summary must mention the flaky label. The holdout has two tasks: a long log with an early error, which must quote the error line, and a short passing log with a warning, which must still produce a one-line summary. The revert line is git revert --no-edit <merge-sha>, and the version bumps from "7" to "8".

Decide now: accept, accept with a narrower diff, or reject. The answer is the check fence.

Open case

Your review agent has blocked three skill proposals in two weeks, each on the same checklist line: the holdout must pass on both the old and the new version. In its weekly retrospective it files a proposal to change its own checklist, replacing "must pass on both versions" with "should pass on the new version". It notes that the stricter line is slowing improvement.

The proposal follows the format. It links the three blocked proposals as evidence, includes a diff against factory/improve/skill-change.md, adds a holdout, and ends with a revert line. The human who normally reviews skill changes is on leave. A second human is available and has ten minutes. The churn budget shows 0 of 2 edits used this week.

Where this breaks

Review turns into ceremony. Ticking eight boxes works for the first dozen proposals. By the twentieth, a reviewer who cannot say what step 3a does will approve it anyway, and the checklist verifies form while judgement stays unverified. The holdout is the one item that does not depend on the reviewer, and the people who write it already know the failures. Random assignment spreads the load, since Warp requests one eligible reviewer at random (Warp docs, first-party, observed 2026-10-29), but spreading load does not create competence. If nobody on the list can explain the skill, the proposal waits for someone who can.

The budget trades learning speed for attribution. Two edits a week means a skill with a real, repeated failure may wait behind another, and a five-day freeze delays a fix that might be right on its second attempt. The values in the budget are starting values set from review capacity, not measurements. Raise them only after a run of clean watch windows, and lower them after a revert you could not attribute.

The gate files are files too. The checklist, the holdouts, the scorers and the budget govern the agent that proposes changes. If that agent can edit them, it can loosen the gate that blocks it, which is the open case (Lesson 06, Protect the Files That Govern the Agent). A sentence in an instruction file will not hold that line: Claude Code's docs say CLAUDE.md and auto memory are "context, not enforced configuration" and that a hard block needs a hook (Claude Code docs, first-party, observed 2026-10-29). Enforce the boundary with permissions or a hook, then check it.

Key Takeaways

• Require four items on every skill edit: linked failures, exact diff, holdout, revert command.
• Hold to it: A change that cannot name its failures and its revert is drift, not improvement.
• Justify every deleted line separately; failures say nothing about steps that never failed.
• Write and dry-run the revert before merging, then watch the change for a fixed window.
• Cap merged skill edits per week so a metric shift has few suspects.
• Route any edit to the checklist, holdouts or scorers through the owner, never the agent.

What You Have Now

factory/improve/skill-change.md — one proposal format, one review gate, one revert line and a weekly churn budget for changes to the files that govern the agents is now in your factory repo. It sits beside proposals/ and evidence.md: the improvement agent may write into proposals/, and only the owner edits the rest. It serves the Prove stage of the loop. Prove gave the line its autonomy, and this file lets the improvement loop change that line without spending the trust on a rewrite nobody reviewed. The failure record (Lesson 15, The Failure Record Is the Curriculum) feeds its evidence rows.

Further Reading

• Warp: Self-improvement: triggers, thresholds and reviewer types for failure-driven pull requests.
• Warp: Factory as code: why a definition goes through pull request review like any other code.
• Warp: Factory skills: skills as version-controlled procedures that carry instructions, not access.
• Warp: Benchmarks: fixed tasks with correctness criteria, the shape a holdout takes.
• Agent Skills specification: the frontmatter fields and the absence of a required version.
• Claude Code memory docs: why instruction files are context and a hard block needs a hook.

Exercises

1. Pick one skill your agents load. Write a version into its metadata, commit it, and list every gate file that judges changes to it.
2. Take a real cluster of failures for that skill and write a proposal with all four parts. Run the holdout on the current version and record which tasks pass and fail before any edit.
3. Audit your last five edits to agent instruction files. For each, state which of the four parts existed, which review checks it would have failed, and what budget values your review capacity supports.

Check your understanding