How Warp improves its own factory

A platform team copied the visible half of an agent factory. They took the dashboard (autonomy, cycle time, cost per PR), pointed a nightly job at the review agent's prompt, and let the job open a pull request whenever the review agent missed something. Nobody measured the review agent first. The job read a handful of misses, rewrote the prompt, and a teammate approved each change because the diff read well. Three weeks in, the review agent began approving migration PRs it used to block: one rewrite had softened a rule that a single noisy miss made look excessive. A bad migration reached production on a Friday and cost a weekend of rollback. The dashboard showed autonomy rising the whole time. This is a composite, not a reported incident: no judge, an editor trusting the missing judge, and a reviewer who saw only the diff. Decide now: which part of a vendor's improvement stack do you copy first, and what evidence must exist before anything may rewrite a prompt?

You build factory/adopt/warp.md — a fill-in plan that borrows Warp's improvement stack: scorer, self-improvement run, memory store and router, each with your own threshold, reviewer and test. It serves the Prove stage of this course's loop (Slice → Gate → Run → Wrap → Prove): Prove decides whether the line earned autonomy, and this file decides which of Warp's published mechanics may spend that evidence to change the line itself, and which you refuse. Warp's docs publish the mechanics in detail and almost no results, which separates what you can copy from what you must produce.

What is actually going on

Warp's factory has a measuring half and an acting half, and the docs describe both. The measuring half is the scorer: an LLM-based eval that checks whether completed runs from selected agents meet criteria you define. It assigns a classification, not a numeric grade, and its fields are the agents to evaluate, judge instructions, a judge model, classifications each with a score, a pass threshold and a sample rate (Warp docs, first-party, observed 2026-10-29). The acting half is the self-improvement run. A failure is a score under the pass threshold. By default, once an agent has 25 distinct unreviewed failures, or its oldest unreviewed failure is seven days old, the run groups that agent's failures into a follow-up run that proposes a fix, and the count is configurable from 1 to 50 (first-party). When the factory is defined as code, that follow-up can open a pull request against the factory definition itself, changing prompts, skills and configuration (first-party).

Two more parts sit beside that loop. Memory writes durable facts after a conversation ends and retrieves them in the background in later runs (first-party). A router picks the model per task (first-party), and benchmarks compare configurations on fixed tasks before a live change (first-party).

Read the stack as one loop with one load-bearing part. Every arrow after the scorer inherits the scorer's errors: a wrong judge fills the queue wrongly, the proposal fixes the wrong thing, and the reviewer approves a diff that reads well. The docs publish no scorer pass rates or judge accuracy, so the polish shows how the stack is wired, not how well it works. The wiring is free to copy. The evidence is yours to produce, which gives the line the whole stack turns on: Adopt the judge before the editor.

In the diagram, look for the one node that every later arrow depends on.

It showed one measuring node and one human node: nothing between them corrects a wrong classification, so the reviewer needs the failed runs, not only the diff.

The console is worth one paragraph of its own, because it is where the governance is visible. In the demo, a scorer's configuration page was read-only with a banner saying the scorer is managed in a repository and changes belong in that repository at a named ref (observed in a public demo, 2026-10-29 frame at 24:40). That single banner is the whole Lesson 06 argument in shipped form: the thing that grades the agents cannot be edited from the surface the agents' work is watched on. The same sidebar held the factory's scorers, its self-improvement queue with an unread count, and its benchmarks, and the factory on screen was one of two the team runs.

The rule and its condition

Adopt Warp's stack in dependency order, and adopt a layer only when you can already produce the evidence that layer spends. Adopt the scorer when an agent already produces runs you can sample. Adopt the improvement run when failures cluster faster than a person can read them and a named person reviews each proposal. Adopt memory when facts are re-derived across runs and each memory traces to its source run. Adopt the router when you own a benchmark built from your tasks. Otherwise refuse that layer and write down why. The table gives Warp's published position on seven fixed rows; the last column is what you must state before the row is yours. Lesson 9, Score the Repo Before You Switch Autonomy On, applies the same idea to the repo.

The tree applies the rule when you have more than two options at each step.

Worked case

A four-person team runs agents on one monorepo with three risk lanes (Lesson 7, Risk Lanes). Lane A is docs and config, lane B is application code, lane C is database migrations and auth. A review agent exists but runs on the same model as the implementer, and nothing scores it. The factory definition lives in the repo and changes go through pull requests (Lesson 6, Protect the Files That Govern the Agent).

Option one: switch the stack on in the order it is easiest to see: dashboard, router, memory, self-improvement. Option two: fill a mechanics table with a lane, a local version, a status and a one-week test per row. Decide now: for which lane and mechanic do you write refuse, and which test gates each row?

The reveal is option two, and the table is the plan.

Illustrative composite. The numbers here are modeled, not measured, and are not Warp's defaults. Assume lane B produces 40 review runs a week, you sample 50%, and 20% of scored runs fail: 20 scored runs, about 4 failures a week. The default count of 25 would take over six weeks to fill, so the seven-day clause fires first and each proposal would rest on about 4 failures. At the default 10% sample rate the same lane yields 4 scored runs and under 1 failure a week.

That arithmetic is the point of the second row. At low volume the age clause, not the count, sets your cadence, so every proposal is a hypothesis from a handful of runs. Hence the named reviewer, the failed runs travelling with the proposal, and the benchmark re-run before merge. Hence also the scorer first: Adopt the judge before the editor. The docs I read confirm two knobs, sample rate and failedRunThreshold; check whether your plan lets you change the age clause.

Lane C gets refusals in three rows: it has the highest cost per wrong approval and the smallest failure volume, so it keeps pinned models, no automated proposals and no memory. Option one fails in order: the router saves money before any judge says whether quality held, memory stores facts from unscored runs, and self-improvement proposes edits a reviewer approves from the diff alone. That is the Friday failure again. The file you write is the table above as a template:

Faded case

Skip to the open case if this is familiar.

A repo has an implement agent and a review agent on the same model, work arriving from a chat thread, and no scorer, benchmark or memory. A manager asks you to turn on self-improvement and a cheaper auto-efficient router this week, because the cost line is visible (first-party).

Pick the one you adopt first, its starting sample rate, and why the other three wait. The scorer docs use 10% as their worked example (first-party).

Open case

A payments-adjacent repo is under change control. Any change to how software is reviewed needs a named, accountable approver recorded in the change ticket, and an approver cannot be picked at random from a group. The factory's review agent has a scorer, and failures cluster weekly. Self-improvement will open pull requests against the review agent's prompt in the factory definition. The compliance lead wants the improvement loop on with the audit trail intact. Write the reviewer configuration you would ship.

Where this breaks

Copying the polish without the evidence layer. The dashboard measures throughput: autonomy is the share of merged PRs that needed no human code push, and cost per PR is a median that splits each run's cost, child runs included, across its PRs (first-party). Neither says the PRs were right, and the cost of improvement runs is unpublished.

A vendor's internal finding does not transfer. "UI tasks run well on one model" was reported without scores, procedure or task distribution. The routing papers report savings on public benchmarks with the authors' own scoring (paper, RouteLLM), and RouterBench exists because routing quality depends on the task distribution (paper, RouterBench). Only a suite built from your tasks answers it.

Memory is not free. Warp states that memory creation and retrieval consume no tokens and add no latency to the active task (first-party). That covers the active task, not review load, stale memories, or a wrong memory superseding a right one. The Mem0 authors report more than 90% token savings against full context, on a conversational benchmark and by their own evaluation (paper), not on coding work. Claude Code's docs add that memory is context, not enforced configuration (first-party).

A memory written from a run that read untrusted text will be retrieved by later runs. The source record is your control.

Key Takeaways

• Copy the evidence layer before the dashboard; wiring tells you nothing about judge accuracy.
• Adopt the judge before the editor.
• Set the failure threshold from your own failure volume, not from Warp's default.
• Name the reviewer in config, then enforce that person with a required-reviewer rule.
• Benchmark your own task class before routing; a vendor's internal finding will not transfer.
• Refuse memory in the migration lane until each memory traces to a reviewed source.

What You Have Now

factory/adopt/warp.md — a fill-in plan that borrows Warp's improvement stack: scorer, self-improvement run, memory store and router, each with your own threshold, reviewer and test. It lives in your repo beside the factory definition, protected like any file that governs the agent, and it serves the Prove stage: it records which of Warp's mechanics earned autonomy in which lane, and which you refused. Lessons 28 to 31 go deeper on the parts this file selects among, and Loop Engineering's locked scorer is the rule behind the scorer-file row.

Further Reading

• Warp: Scorers: the fields and classification model behind the judge.
• Warp: Self-improvement: trigger, failedRunThreshold and reviewerType in full.
• Warp: Benchmarks: suites, trials and comparing configurations before a live change.
• Warp: Custom routers: complexity-based and rule-based routing.
• Warp: Agent Memory: stores, access modes, traceability and auditability.
• RouterBench: why routing quality is measured against a task distribution.

Exercises

1. Create factory/adopt/warp.md — a fill-in plan that borrows Warp's improvement stack: scorer, self-improvement run, memory store and router, each with your own threshold, reviewer and test for one repo, with all four sections and a status of adopt, adapt or refuse for each lane.
2. Run the seeded-defect test for your scorer and record how many of ten defects it flags; under your threshold, rewrite the judge first.
3. Build a ten-task benchmark from last month's merged PRs in one task class and compare two model configurations on quality and cost. Record it in the Router test field.

Check your understanding