Build the improvement layer in six weeks

You know this team. They finished the reading and shipped none of it. Their repo now holds three things, each reasonable on its own. A notes file the agents append to after every run, which nobody has opened since the week it was created. A router that sends reviews to one model and migrations to another, with no benchmark behind either choice. A nightly job that rewrites the agents' own prompts and opens pull requests. The queue for those pull requests is now the longest in the repo, because the only people who can judge them are the engineers the agents were meant to free. The notes contradict each other, the router cannot say whether it saved a cent, and the prompt edits cannot be compared with anything because nobody wrote down what better meant first. Humans now carry the cost of three systems and have removed no work from the line. Decide now: which of the three do you switch off first, and what must you see before you switch it back on?

You leave with factory/improve/rollout.md — a six-week sequence with a gate at the end of each week, the file each week produces, and the condition that stops the rollout. It serves the Prove stage of this course's loop (Slice → Gate → Run → Wrap → Prove): Prove decides whether the line earned autonomy, and the rollout makes sure the evidence for that decision exists before the improvement loop starts spending it on changes to the line itself.

What is actually going on

The three half-layers fail for one reason: each is an open loop. A notes file with no record of which entries a run used cannot say which to prune. A router with no benchmark cannot say whether a change helped. An improvement job with no named cluster proposes edits nothing can grade. They were started in the wrong order, and each consumes a signal an earlier layer should have produced.

The sequencing principle has three clauses. Instrument before improving: the run log gains cost and a line naming the memories and skills each run used, and a scorer classifies a sample of one agent's runs. Improve the class you can measure: the first change targets one failure cluster on one agent, because a cluster is the smallest unit a holdout can grade. Widen only after a revert: scope grows past the first agent in W6, and only once one merged change has been reverted and the holdout has returned to its old count.

Every input already exists: Lesson 09, Score the Repo Before You Switch Autonomy On, gave you a baseline. Lesson 14, Measure Outcomes, Not the Share of AI Code, gave you the outcome metrics. Lesson 15, The Failure Record Is the Curriculum, gave you the past failures that W2 and W5 draw from. Lesson 28 left the loop map that W1 adds rows to, Lesson 29 the skill-change format W3 applies, Lesson 30 the memory schema W4 fills, and Lesson 31 the task-class table and benchmark card W5 changes by one row. Whatever Lesson 32 left in factory/improve/ joins the W6 review.

Two numbers are borrowed, then overridden. A scorer assigns a classification, not a numeric grade, and a 10% sample rate evaluates about one in ten runs (Warp docs, first-party, observed 2026-10-29), so twenty classified runs at that rate need about two hundred runs. Choose the rate from your volume. The vendor's default trigger for a follow-up proposal is 25 distinct unreviewed failures, or the oldest reaching seven days, with failedRunThreshold configurable from 1 to 50 (Warp docs, first-party, observed 2026-10-29). That default fits a factory with volume. W2 sets yours at three attributable failures in one cluster, because the goal of W2 is one reviewable proposal, not coverage.

Look for the dashed edge in the diagram: every week has one exit to the next gate and one to the stop.

Each week exits only through its gate or to the stop. No arrow skips a gate, and none returns to an earlier week unless the log gives a reason.

The rule and its condition

Use the gated sequence when your agents produce at least one failure a week that you can trace to a skill, a memory entry or a routing choice, and one named reviewer can decide a proposal in fifteen minutes. Otherwise do not build the layer: fix the line by hand using the failure record from Lesson 15, and revisit when attributable failures exist. Inside the sequence the rule is mechanical. A week ends when its count or timestamp says so. Earn each week with a number, or stop.

The table compares the sequence with the pattern from the opening, under one condition: a single reviewer with finite hours.

The tree covers a gate that fails. One retry is allowed, with one variable changed. A second failure forces you to name a cause, or the rollout ends. This is the discipline of the ratchet in Loop Engineering, Lesson 3: you do not move the measure to make a week pass.

One path leads to the next week and one to the end. Every other branch is a logged repair.

Worked case

Your repo is ledger-api, a TypeScript service with three agents: triage, implement and review. W1 put a scorer on the review agent at a low sample rate and added the used-skills line to the loop map. W2 produced cluster C-001: the review agent labels style nits blocking, which bounces pull requests over rename requests. Four failures, each traced to skills/review-severity/SKILL.md, which never defines blocking.

Constraints: one reviewer, the platform lead, with fifteen minutes per proposal. The skill has one owner and one version (Lesson 11, One Owner and One Version per Skill). The agent that writes the proposal never holds the merge key (Lesson 04, The Author Never Holds the Merge Key). The scorer and everything under factory/improve/ are protected paths (Lesson 06, Protect the Files That Govern the Agent). This is the file as you copy it; yours has W1 and W2 logged.

Illustrative composite. The hours per week, run ids, holdout counts and timestamps in this case are modeled planning figures that show the shape of the records. They are not measured on a real repo. Replace the hours with your own after W1.

Three shapes for the first proposal:

• A. Rewrite the whole review skill.
• B. Add a six-line severity section to the existing skill.
• C. Lower the review agent's blocking flag for every comment type.

Decide now: which shape do you accept for P-001?

B. One cluster licenses one change, and this week's gate measures whether the reviewer can decide, so the proposal must be readable in fifteen minutes. B adds six lines, so the diff is the review. It also gives the holdout a clean question: on twelve past review runs the proposer never saw, labelled by a human, how many nits does the old skill mark blocking, and how many does the new one? Lesson 05, Holdouts Need Exact Figures, is why the answer is a count, not "mostly better".

The spec requires no version field and gives metadata as an arbitrary string-to-string map (Agent Skills specification, first-party, observed 2026-10-29), so owner and version go there. One machine step produces the numbers; one human step, the only one with a clock, consumes them.

The reviewer works a fixed checklist, fifteen minutes in total, starting from opened:

1. Scope, 1 min. The diff touches the skill file and proposals/P-001.md, nothing under a protected path.
2. Evidence, 3 min. Open the four run ids. Each shows a nit labelled blocking, and each traces to the missing definition.
3. Change, 3 min. Six added lines. The rule fits in one sentence. No unrelated edits.
4. Holdout, 4 min. Counts are exact, before and after, on runs the proposer never saw. No real defect was downgraded.
5. Revert, 2 min. The revert line names one commit and the holdout count to expect after reverting.
6. Decision, 2 min. Accept or reject, and write the time.

Record the result in w3-log.md:

Option A costs you the gate. A rewrite cannot be read in fifteen minutes, so the reviewer rubber-stamps it or overruns, and a regression cannot be traced to a paragraph. Option C looks cheaper and is worse: one flag changes behaviour on every comment type, the holdout covers only nits, and a real defect labelled nit now merges.

Stop looks like this. Suppose the reviewer needs 22 minutes on P-001 and 19 on P-002. That is the W3 stop condition, so P-003 waits while the tree asks for a cause. One is available: both proposals bundled several changes. That is a scope cause, so you require one hunk per proposal and retry. If P-003 is decided in eight minutes, W3 passes. If it overruns with no cause you can name, you append STOPPED at W3, <date>, no cause named to the log and W4 never starts.

Faded case

Skip to the open case if this is familiar.

Your repo is docs-site. The triage agent has a scorer at a low sample rate, and W1 has run for two windows. Window one: 24 runs classified, 5 failures. Window two: 31 classified, 4 failures. The count part of the gate passes both times. The naming part does not. Every failure sentence reads "ticket had no reproduction steps, agent guessed". The triage skill already tells the agent to ask for reproduction steps; the failing runs came from chat threads where nobody could answer. No failure implicates a skill, a memory entry or a routing rule, so W2 has nothing to cluster. The review agent on the same repo ran about as often in the same two weeks, and nobody has scored it.

Illustrative composite. The run and failure counts are modeled, not measured.

The tree has run once already: the first miss led to a retry with a longer window. The second miss is where the tree asks for a cause, and here the cause is known. Match it to a branch before you move. Do not choose by effort.

You have three moves. Change the scorer so it fails only runs the agent could have avoided. Move the scorer to the review agent. Stop the program. Pick one and write the log line.

Open case

A solo maintainer runs a command-line tool. A coding agent triages issues and drafts fixes. The maintainer has two hours a week for everything to do with the agent, review included, and no second reviewer. About a dozen issues arrive each week, and three or four fixes merge. The agent appends to a sixty-line memory file, and one model handles all work. The maintainer wants the improvement layer.

Write their rollout.md. Say which weeks they run and in what order, how many calendar weeks that takes, what they defer, which gates stay exactly as written and which numbers they change, and the line that stops the rollout.

Where this breaks

The program becomes the work. The modeled estimate is 26 hours across six weeks, and the dashboard will not say whether that paid off: run counts include evaluation, benchmark and self-improvement runs, so more runs with flat PRs can be your own measurement activity (Warp docs, first-party, observed 2026-10-29). Every dashboard line in W6 must name the decision it triggers. A line that triggers nothing is a cost.

The reviewer is the ceiling. Each week adds a review obligation: proposals, memory curation events, routing changes. In Agent Fleets, review throughput is the ceiling, and no schedule here raises it. If W3 runs long twice, the answer is smaller proposals or a higher threshold, never reviewerType: none.

The gates are small samples. Twenty runs, a cluster of three and a holdout of twelve justify an attempt a human can inspect. They do not prove an improvement. The vendor team has said publicly that an improvement agent can produce a worse procedure, and published no figures. The gates bound how much goes wrong, the revert line caps it, and nothing in this sequence makes the result statistically sound. Cluster evidence is run-log text that came from tickets and diffs; the proposer reads it as data, and the protected paths keep it from rewriting the gates.

Key Takeaways

• Order the layers by dependency: instrument, cluster, propose, remember, route, widen.
• Gate every week on a count or a timestamp, never on a feeling.
• Treat the vendor default of 25 failures as a reference point, then set your own threshold.
• Time the reviewer: a proposal that cannot be decided in fifteen minutes fails W3.
• Remember the rule: Earn each week with a number, or stop.
• Widen scope only after a revert has returned the holdout to its old count.

What You Have Now

factory/improve/rollout.md — a six-week sequence with a gate at the end of each week, the file each week produces, and the condition that stops the rollout. It lives in your repo beside the loop map, scorer, proposal, memory and routing files it sequences, under the protected paths of Lesson 06, so the improvement agent can read it and cannot edit it. It serves the Prove stage: each line in its status log is evidence that the line earned the next step, and the weeks that pass become the record you present when you ask for more autonomy.

Further Reading

• Warp: Self-improvement: the default trigger you override in W2.
• Warp: Scorers: classifications, pass threshold and sample rate for W1.
• Warp: Benchmarks: suites and trials behind the W5 card.
• Warp: Measure and improve: the W6 dashboard metrics and caveats.
• Agent Skills specification: why owner and version live in metadata.
• Claude Code memory: a load cap that frames W4.

Exercises

1. Create factory/improve/rollout.md — a six-week sequence with a gate at the end of each week, the file each week produces, and the condition that stops the rollout. Copy the table from the worked case, replace the hours with your estimates, and name one reviewer.
2. Run W1 on one agent. Set the sample rate so about twenty runs a week get scored, capped so that no more than every run is scored. Log the date, the classified count and the named failure. If the gate fails, walk the tree.
3. Take a repo with five agents. Choose which agent W1 should instrument, and defend the choice in three sentences against the strongest case for instrumenting the agent that costs the most.

Check your understanding