Route by task class, then prove the routing

A factory ran triage, spec, implementation and review on the most expensive model it could reach, for a full quarter. The reason on file was one word: quality. When finance asked for a cost review, the breakdown showed that the two highest-volume lanes, mechanical migrations and documentation updates, were also the lowest-ambiguity work in the building, and both were billed at frontier prices. The obvious question was whether a cheaper model would have done those tasks as well. Nobody could answer it. No one had ever run two models on the same tasks, so the review ended in a guess instead of a saving. A second team made the opposite move and put its reviewer on a cheaper model to cut spend. For a month the review comments got vaguer, "consider simplifying this" in place of a failing input, and nobody connected that to the swap until a defect shipped that review should have caught. Decide now: when one model choice repeats across every run of a recurring job, what must you hold in your hands before you change it in either direction?

The deliverable is factory/improve/routing-rules.yaml — a task-class table with the model per class, the classes that may never be downgraded, and the benchmark card that earns each rule. It serves the Prove stage of this course's loop (Slice → Gate → Run → Wrap → Prove): Prove decides whether a line earned autonomy, and handing a class of work to a cheaper model is autonomy spent on a bet, so it needs the same kind of evidence as any other promotion.

What is actually going on

A factory sends every work item through the same sequence: triage, planning, implementation, review, with a pull request at the end and no merge on its own authority (Warp docs, first-party, observed 2026-10-29). Each stage is an agent, and each agent has a model. A model choice is therefore a standing multiplier: an error repeats on every item of that kind.

Platforms now make the choice explicit per agent. You set a model and a harness (the Warp Agent, Claude Code or Codex) for each one, and the docs say what each role optimizes for: triage wants research and evidence gathering, spec wants precise technical writing, implement wants coding strength, and review should run on a different model or harness from implement so the two don't share blind spots (Warp docs, first-party, observed 2026-10-29). That last constraint has nothing to do with price.

Choosing per item is a second layer. A complexity-based router sorts each task into easy, medium or hard and maps each level to a model. A rule-based router pairs a natural-language classification prompt with a model (Warp docs, first-party, observed 2026-10-29). Built-in options such as auto-efficient and auto-genius trade cost against capability for you (Warp docs, first-party, observed 2026-10-29). The vendor team publicly recommended starting on auto models (vendor-claim, no figures published). That is a starting point, not evidence about your tasks.

Research agrees the savings exist and says nothing local. RouteLLM trains routers on human preference data to pick between a strong and a weak model, and reports cost reductions of over 2 times without sacrificing response quality on public benchmarks (RouteLLM, paper, observed 2026-10-29). RouterBench exists because routing quality is measured against a task distribution (RouterBench, paper, observed 2026-10-29). Both are the authors' own evaluations on public sets. Which model wins on your migrations is answerable only by trials of your tasks under each configuration, judged by your correctness criteria (Warp docs, first-party, observed 2026-10-29).

Three parts: a classifier names the class, a rule table maps class to model and harness, and benchmark and scorer evidence is the only thing allowed to edit the table. In the diagram, the dotted edge is the only path by which the table changes.

The diagram showed that the run does not talk to the table directly. Evidence does, and only as a proposal a human reviews.

The rule and its condition

Use a rule table when a class has a benchmark card and enough volume to pay for building it; otherwise use an auto model for classes that don't gate the merge, and pin the model you already run for the ones that do. Route by task class, never by convenience. "This one looks quick" and "this model is already configured" are convenience. A class is a recurring kind of work with a stable distribution, a stable correctness criterion and one owner. A working test for a class: two people given the same item assign it the same class, and its failures have one owner who reads the scorer cards. Rank candidates for a first card by volume and low ambiguity, skipping any that gate the merge. Prefer matches you can compute (touched paths, work source, risk lane from Lesson 07, Risk Lanes) over matches a model must judge; a classification prompt has its own error rate.

Two conditions are hard. A rule exists only with a benchmark card behind it: a non-default model in the table without a card is a guess stored in a config file. And classes that gate the merge, review, verification and security review, may not be downgraded without a holdout that proves no loss. The asymmetry is deliberate. A weaker implementer is caught by the reviewer, so its failures are loud. A weaker reviewer fails by saying less, and nothing downstream complains. Downgrade where mistakes are loud, never where they are quiet.

Worked case

You run a six-class factory: triage, spec, implement-feature, implement-fixup, docs and review. Mechanical migrations fall under implement-fixup. All six classes run on one frontier model. You are asked to cut the bill, and you will write factory/improve/routing-rules.yaml — a task-class table with the model per class, the classes that may never be downgraded, and the benchmark card that earns each rule to do it. Constraints: one review per PR, on a different model or harness from implementation, and no comparison data.

Illustrative composite. Volumes, token counts and prices are modeled, not measured. Blended price per million tokens: frontier $30, mid $6, small $1.50.

Class	Runs per month	Tokens per run	Monthly cost, all frontier
triage	1,200	150k	$5,400
spec	150	300k	$1,350
implement-feature	120	1.5M	$5,400
implement-fixup	400	600k	$7,200
docs	500	400k	$6,000
review	1,020	500k	$15,300
Total			$40,650

Options: put everything on auto-efficient; drop review, the largest line, to the mid model; or move only high-volume, low-ambiguity classes to cheaper models, one at a time, after each earns a card. Decide now: which class do you try to cheapen first, and which do you refuse to touch?

The third option wins, and review is the class you refuse. Reasoning, in order:

1. Review is the biggest line in that table and the tempting first cut: moving it to mid would save roughly a third of the monthly bill on paper. Its failure is silent, as in the opening, and it is the gate. Pin it, on a different model and harness from implementation.
2. Docs and fixup combine the highest volume with the lowest ambiguity, and a failed one is caught by review. Plausibly equal is not proven equal, so each gets a card.
3. Triage, spec and feature stay put with pending cards. Writing a card is cheap; promoting a model waits.

The file is shown in full, in the course's own format. Warp loads routers and benchmark suites from routers/by-task.yaml and benchmarks/<suite>/suite.yaml (Warp docs, first-party, observed 2026-10-29); their field names are in the published schemas, which this file maps onto rather than copies.

# factory/improve/routing-rules.yaml# may_downgrade: true = cheaper allowed once its card is earned; false = only via a no-loss holdout in a human-approved PRversion: 1default_class: implement-feature   # unmatched work gets the safe route, never the cheap oneclasses:  triage:    match: "bug report or question; no code change requested yet"    model: frontier-a    harness: warp-agent    rationale: "research and evidence gathering; errors poison later stages"    benchmark: bench-triage-v1    fallback: frontier-b    may_downgrade: true  spec:    match: "triaged item that needs a written spec"    model: frontier-a    harness: claude-code    rationale: "precise writing; implementation inherits spec errors"    benchmark: bench-spec-v1    fallback: frontier-b    may_downgrade: true  implement-feature:    match: "approved spec that adds behavior or a public interface"    model: frontier-a    harness: claude-code    rationale: "coding strength on long, ambiguous work"    benchmark: bench-feature-v1    fallback: frontier-b    may_downgrade: true  implement-fixup:    match: "review comment, CI failure, or mechanical migration under 200 changed lines"    model: mid-a                   # was frontier-a; flipped in the PR that earned the card    harness: claude-code    rationale: "narrow, well-specified edits"    benchmark: bench-fixup-v1    fallback: frontier-a    may_downgrade: true  docs:    match: "diff touches only docs/ or *.md"    model: small-a                 # was frontier-a; flipped in the PR that earned the card    harness: claude-code    rationale: "lowest ambiguity, highest volume"    benchmark: bench-docs-v1    fallback: mid-a    may_downgrade: true  review:    match: "every PR, before human review"    model: frontier-b    harness: codex    rationale: "gates the merge; differs from implement in model or harness"    benchmark: bench-review-v1    fallback: hold                 # never fall back to the implement agent's model    may_downgrade: falsecards:  bench-fixup-v1:    class: implement-fixup    baseline: frontier-a    candidate: mid-a    task_set: "60 fixups from merged PRs of the last 90 days, frozen 2026-10-29"    holdout: "12 of the 60, unseen by anyone who edits prompts or skills"    correctness: ["CI green", "comment resolved per scorer fixup-judge", "diff stays in files the thread names", "no test deleted or skipped"]    repetitions: 3    cost_measured: "tokens x price per trial, child runs included"    decision_rule: "adopt if candidate holdout passes >= baseline holdout passes AND candidate passes >= baseline passes - 2 on all 60 AND zero severity-high failures"    rerun_when: ["model version changes", "match rule changes", "every 90 days"]    status: earned  bench-docs-v1:    class: docs    baseline: frontier-a    candidate: small-a    task_set: "40 docs diffs sampled from merged PRs, frozen 2026-10-29"    holdout: "10 of the 40"    correctness: ["links resolve", "no claim the code does not support", "style checks pass"]    repetitions: 3    cost_measured: "same as bench-fixup-v1"    decision_rule: "same as bench-fixup-v1, with 2 replaced by 1"    status: earned  bench-triage-v1: { class: triage, status: pending }  bench-spec-v1: { class: spec, status: pending }  bench-feature-v1: { class: implement-feature, status: pending }  bench-review-v1: { class: review, status: pending }invariants:                        # a CI check on this file fails the PR if any is false  - "for every implement class: review.model != its model or review.harness != its harness"  - "a class may differ from its card baseline only when the card status is earned"  - "a class with may_downgrade false changes model only in a PR with a holdout result attached"

The card does the work. The task set comes from merged history, not from the people tuning prompts, and is frozen so later edits cannot flatter either side. The holdout is a slice nobody tunes against. Repetitions matter because one trial is one draw from a noisy process (Warp docs, first-party, observed 2026-10-29). The decision rule is written before the trials run, with exact counts (Lesson 05, Holdouts Need Exact Figures). The scorer and task files are protected paths (Lesson 06, Protect the Files That Govern the Agent), so the run proposing a route change cannot edit the yardstick it is judged by.

Illustrative composite. The same modeled prices apply. Docs card: 40 tasks x 3 repetitions x 2 configurations = 240 trials. Baseline runs cost $12 each (0.4M tokens at $30), 120 of them: $1,440. Candidate runs cost $0.60 each, 120 of them: $72. Judge calls at $1.50 each, 240 of them: $360. Card cost: $1,872. Monthly saving on docs: $6,000 - $300 = $5,700. Break-even: about 10 days. Fixup card: 60 x 3 x 2 = 360 trials; $3,240 + $648 + $540 = $4,428 against a monthly saving of $7,200 - $1,440 = $5,760: about 23 days. After both flips the modeled bill is $24,870, and review is $15,300 of it, about three fifths.

That last share is not a prompt to cut review. It grew because the other lines shrank. Watch cost per PR and autonomy together, as the dashboard defines them: median cost per PR with child runs included, a comparison metric rather than a billing figure, next to the share of merged PRs that needed no human code push (Warp docs, first-party, observed 2026-10-29). A cheaper lane whose autonomy drops has moved its cost onto people (Lesson 14, Measure Outcomes, Not the Share of AI Code).

The rollout has three moves, in order. First, shadow mode: the candidate runs beside the baseline on live items of that class, its output discarded, and a scorer classifies both (Warp docs, first-party, observed 2026-10-29). Second, one class at a time, docs first, then fixup, because two flips in one PR make a regression unattributable. Third, the review rule: after each flip, the invariants confirm the reviewer still differs from every implement class. The rejected option, review on mid, saved $12,240 a month on paper and bought the opening failure. Cheaper lanes also change how many agents you can afford, the starting input of Agent Fleets, The Cost Model Picks Your Topology.

Faded case

Skip to the open case if this is familiar.

Add a seventh class, implement-migration: schema migrations against production tables, with a staging dry-run and review ahead of any human merge. The vendor team has publicly shown a custom rule that sends database migrations to a different model (vendor-claim, no figures published). You build bench-migration-v1: 50 tasks from past migrations, one trial each against a staging copy, same correctness criteria for both configurations. The baseline model passes 50 of 50. The mid model passes 50 of 50.

Illustrative composite. Modeled: 30 migration runs per month at 0.8M tokens each; frontier $30 and mid $6 per million tokens. Saving from the flip: 30 x 0.8 x ($30 - $6) = $576 a month. With zero failures in 50 independent tasks, a rough 95% upper bound on the failure rate is 3 in 50, or 6%. Assume the staging dry-run and review together stop 19 of 20 bad migrations.

The row would match diffs under migrations/, run on claude-code, and fall back to frontier-a. The card cannot answer how often a clean-looking migration damages data in production, because 50 tasks drawn from past successes rarely include that case. One decision is left to you: flip the row to the mid model, or hold it. Work out the number that decides it, then check.

Open case

Your foreman agent keeps a thread open for days. It reads a ticket, dispatches triage, spec, implement and review children, answers humans in the thread, and revises its plan as work lands. The router is the rule-based kind and chooses a model when the conversation starts. The thread opened with a two-sentence request to look into a flaky login test, was classified as easy, and went to the small model. By day two it is coordinating a migration across four repositories with eleven child runs. It drops instructions humans post in the thread and starts children out of order. The cost split by model shows the small-model lane as the cheapest line in the factory, while PRs from these threads need the most human pushes before merging. No benchmark covers foremen. Write the foreman's row.

Where this breaks

Cards overfit and drift. A suite you wrote yourself rewards the tasks you thought of, so sample from merged history and keep a slice nobody tunes against. Then watch the class: a docs lane that starts receiving API reference rewrites is a different class wearing the old name. Scorer failures grouped by class give the early signal. By default, self-improvement groups an agent's failures into a follow-up pull request after 25 distinct unreviewed failures, or when the oldest is seven days old (Warp docs, first-party, observed 2026-10-29). A proposed routing change from that path still needs your card.

Routers choose once. The docs state that Warp chooses a model when you start a conversation and keeps using it for the rest of that conversation, with very short, trivial prompts not locking in the choice (Warp docs, first-party, observed 2026-10-29). A thread that grows keeps the model picked at its smallest. Match long-running roles by agent.

Fallbacks are silent routing changes. Outages arrive on days you didn't plan, and a quiet failover to another model is a rule change with no card. If review fell back to the implement agent's model, the different-model rule would break without anyone deciding it. The row says hold; the cost is a queue of waiting PRs. Failing over to a second provider also sends your code to another vendor under another credential, a security decision rather than a routing default.

Key Takeaways

• Route by task class, matching on paths and sources before prompt text, never by convenience.
• Downgrade where mistakes are loud, never where they are quiet.
• Earn every non-default model with a card whose decision rule you wrote before the trials.
• Pin review to a different model or harness than implement, and hold the PR when no such fallback exists.
• Re-run cards when models, match rules or task distributions change.

What You Have Now

You hold factory/improve/routing-rules.yaml — a task-class table with the model per class, the classes that may never be downgraded, and the benchmark card that earns each rule. It lives in your factory definition's improve/ directory under protected-path review, next to the scorers and suites whose evidence it spends, and its cards are re-run on the schedule they name. It serves the Prove stage: each card proves a line earned a cheaper model, and the invariants keep gate classes from being cut for savings alone.

Further Reading

• Custom routers: complexity-based and rule-based routing; pick-once behavior.
• Model choice: what the four auto models trade.
• Benchmarks: suites, correctness criteria, trials, repetitions.
• RouteLLM: learned strong-versus-weak routers on public benchmarks.
• RouterBench: why routing quality is measured against a task distribution.

Exercises

1. Label 30 days of runs with the six classes, cost each class, and name the two largest lines that do not gate the merge.
2. For the larger, write the card: 20 tasks from merged history, frozen, a holdout slice, correctness criteria, and a decision rule with exact counts, before any trial. Run it in shadow mode.
3. Write the invariants as a CI check that fails when review matches an implement class. Then design the foreman row and its card for your factory.

Check your understanding