Measure Outcomes, Not the Share of AI-Written Code

An analytics vendor took two years of telemetry from 22,000 developers on more than 4,000 teams and compared each organisation's quietest months of AI use with its busiest. The throughput half looked like a win: merged pull requests per developer rose 16.2% and finished tasks per developer rose 33.7%. The quality half did not. Incidents per merged pull request rose 242.7% in the same comparison (Faros, vendor telemetry, observed 2026-10-28). A manager who watched only the first half would have asked for more of it. A manager who watched the second would have learned about it from the pager, weeks after the merges were counted and celebrated. The vendor says its own data is correlational, so it cannot prove the tools caused the incidents. It does show what a board with one number on it cannot see. A second report from the same vendor found that merges skipping review grew by 31.3% in the first report and by 76.3% in the next (Faros, vendor telemetry, observed 2026-10-28). Decide now: if your own board showed merged PRs per week climbing every month, which other numbers would have to sit on the same row before you let anyone call that a win?

This lesson gives you factory/scoreboard.md - six metrics with definition, denominator, guardrail pairing and the tripwire that pauses a lane. It serves the Prove stage of the course loop (Slice, Gate, Run, Wrap, Prove): the stage where you find out whether the line you built produces work you would defend, using numbers that can embarrass you.

What is actually going on

A throughput number counts events at the front of the line. Harm shows up at the back, later, and on a different dashboard. That gap is the whole mechanism. When agents raise the number of pull requests entering review, the number of humans and gates able to read them stays where it was (Agent Fleets, lesson 11 covers that ceiling). Something has to give, and the cheapest thing to give up is the reading. Unreviewed changes then reach production, and the incidents arrive after the merge counters have already been banked.

Two details from the later Faros data change how you build the board. In that data the incident-to-PR ratio rose by only 14.5%, a small fraction of the earlier jump, yet monthly incidents rose by 125.4% (Faros, vendor blog on its later data, observed 2026-10-28). A rate can look calmer while the count the on-call engineer feels gets worse, because the volume in the denominator grew. You need a rate and a count side by side. The second detail is the source: a vendor's telemetry, compared within organisations, with causality not established. Read it as a pattern your own board should test, not as a law.

The independent studies point the same two-sided way. DORA's 2025 study of about 5,000 technology professionals found AI adoption now associated with higher delivery throughput and still associated with more instability, and it frames AI as an amplifier of whatever delivery system you already run (Google Cloud DORA, survey and correlational, observed 2026-10-28). Time saved on writing gets spent on checking.

The third piece explains why you cannot trust how fast the work feels. In METR's randomised trial, experienced open-source developers forecast a 24% time saving, believed afterwards that they had saved 20%, and were measured taking 19% longer (METR, arXiv preprint, observed 2026-10-28). That trial used a small group of developers on early-2025 tools, and METR itself says not to read it as today's effect. The durable finding is the gap between belief and measurement, not the 19%.

Trace one merged PR through the diagram below: the first box is the only one a throughput dashboard shows.

Each arrow is a delay or a hand-off to someone else's metric, which is why a board built from the first box alone reports success right up to the page.

The rule and its condition

Use a paired board when agent-authored changes can reach a branch that other people deploy from. Show no throughput figure without its guardrails on the same row, write down the denominator of every ratio, and judge a lane by its guardrails before its volume. Otherwise, while agents only draft and a human authors every merge, throughput plus revert rate is enough, and the full board is overhead.

The condition is checkable in a minute: count last month's merges to the default branch that came from an agent identity in the identity ledger (lesson 12 of this course). Above zero, the board applies.

Count what came back, not who typed it. The share of AI-written code is not on that board, and the table shows why.

The figures people quote come with their limits attached. Google reports that 75% of its new code is now AI-generated and approved by engineers, self-reported, with the definition undisclosed (Google, self-reported, observed 2026-10-28). Uber reports that more than 70% of its PRs are attributed to agents, which is attribution rather than autonomous authorship, again without a definition (Uber, self-reported, observed 2026-10-28). Neither tells you what came back. Uber does say it tracks cost per merged PR and revert rate for each managed agent, but publishes no values. A Microsoft field study found 24.0% more merged PRs per engineer per day after agent CLIs arrived (Microsoft researchers, arXiv preprint, first-party, observed 2026-10-28) and says quality and incident impact were not measured. A careful output number is still only an output number.

Three branches decide a weekly read, so a tree beats a sentence. Follow it from the top for one lane.

Only one exit pauses a lane, and it is gated by a number written down before the week began. The hold exit exists so that a small sample never pauses or clears a lane.

Worked case

Illustrative composite. Larch Freight is invented. Eight engineers share one repo, dispatch-api. September had no agent lane. In October two lanes came on: dependency bumps and test writing. The old dashboard showed merged PRs per week and the share of agent-written PRs. Each window is four weeks, and for this exercise every October merge counts as old enough to read.

Measure	September	Human PRs, Oct	Dependency lane	Test lane	Whole repo, Oct
Merged PRs	160	132	60	40	232
Merged PRs per engineer-week	5.0	n/a	n/a	n/a	7.25
Incidents traced to a merged PR	6	4	9	1	14
Incidents per 100 merged PRs	3.75	3.03	15.0	2.5	6.03
Merges with no approval and no review gate	8 (5.0%)	4 (3.0%)	28 (46.7%)	3 (7.5%)	35 (15.1%)
Reverted within 14 days	5 (3.1%)	3 (2.3%)	12 (20.0%)	1 (2.5%)	16 (6.9%)
Lane spend, kept PRs	n/a	n/a	$1,260 over 48 kept = $26.25	$480 over 39 kept = $12.31	n/a
Private eval, 60 tasks	43 passes	n/a	n/a (no dependency-bump tasks in the set)	n/a	44 passes

The old dashboard read: merged PRs per engineer-week up 45%, agent share of merges 43%. The forecast written on 1 October said throughput up about a quarter and incidents per 100 merged PRs flat. Measured: throughput up more than forecast, incidents per 100 merged PRs up from 3.75 to 6.03.

The tech lead has three options. Option A keeps the dashboard and adds a line chart of agent share. Option B adds one number, total incidents per month, to the dashboard. Option C builds the six-row board and lets tripwires decide. Both lanes are live, the team is small, and a bad dependency bump can break a release.

Decide now: which lane, if any, do you pause, which number says so, and what does the number you did not look at tell you about your eval?

Reveal. Pause the dependency lane and keep the test lane. Start with the forecast, because it is written down and cannot be argued with afterwards: you expected more output and flat incidents, and you got more output than predicted and a clear rise in incidents per PR. Then split by lane, because the whole-repo numbers average a bad lane with a healthy one. The dependency lane's incident rate is several times the human rate, nearly half its merges had no approval and no review gate, and its revert rate is far more than twice the human rate. Three tripwires fire at once. The test lane trips neither the incident nor the revert tripwire, and it costs roughly half as much per kept PR. Its unreviewed-merge share sits a little above September's, so log a watch flag and read it again next Monday. Cost per kept PR confirms rather than decides, because reverts silently double what a dependency merge costs.

The eval row is the quiet lesson. The private eval moved by one task between the two months, which is inside the noise floor (Loop Engineering, lecture 3, on the noise floor), so it says nothing changed. It also contains no dependency-bump tasks, so it could not have caught this lane. Eval flat while reverts climb means the set is stale: add the dozen reverted bumps to it this week, with a hidden holdout the agents never see (AI Solutions Architect, lesson 10 covers evals as the contract). The spend tiers from Agent Fleets, lesson 3 (the cost model) and Loop Engineering, lecture 4 set the budget the cost row is compared against, and AI Solutions Architect, lesson 11 covers the cost model behind it.

This is the file you write. Copy it into factory/ and fill the owner and the dates.

The rejected options cost real things. Option A would have kept both lanes running and charted a rising agent share as progress while the dependency lane kept reverting. Option B shows the count moving, but a count alone cannot say which lane produced it, or whether risk grew or only volume did, and the later Faros data shows the rate and the count can disagree (Faros, vendor blog, observed 2026-10-28). The board costs a Monday morning per lane and one person who owns the file.

Faded case

Skip to the open case if this is familiar. Same board, one decision left for you. A second invented team picks a tripwire for metric 3, the unreviewed-merge share.

Illustrative composite. Quill Billing is invented. Before its test-writing lane started, its weekly unreviewed-merge share ran between 2% and 4%. Over the first six weeks with the lane, the weekly values were 3%, 4%, 3%, 6%, 9% and 14%. Merged PRs per week rose from 40 to 90 across the same six weeks. The lead is choosing between three tripwires: (a) pause the first week the share is above the band; (b) pause when the share is above the band two weeks running; (c) pause at a fixed 20%.

Pick one, say in which week it fires, and say why the other two are worse. Add the extra rule the board needs for protected paths.

Open case

Illustrative composite. A CTO at an invented logistics firm wants one slide for the board reporting on the agent program. She proposes three headline numbers: percent of code written by AI, merged PRs per week, and developer satisfaction. You have the incident tracker, git history with agent identities, review approvals and a year of reverts. Her deadline is Friday.

Write the board you would put on that slide. For each of her three numbers, say whether you keep it, change it or drop it, and what replaces it. Name the denominator of every figure you print, and say what you would write on Monday, before the first lane runs, so that you are not grading yourself against a memory. The quiz grades your answer.

Where this breaks

The evidence is correlational. Faros is a vendor selling analytics, DORA is a survey, and the Microsoft study measures output in a single company's own rollout. None of them shows that AI caused the incidents, and your board will not either. It tells you what your repo did after a lane came on, compared with the same repo's human PRs in the same window. That is enough to pause a lane, not enough to publish a cause.

Task stopwatches stopped working. METR's follow-up study met selection effects and concurrent agent use that broke time-per-task measurement, and METR judged the data an unreliable signal and announced a redesign (METR, observed 2026-10-28). Use METR for the perception gap and nothing more. This is why the board counts shipped outcomes.

Two inputs have thin data. Public benchmarks decay. OpenAI stopped reporting SWE-bench Verified after auditing 138 hard problems and finding 59.4% had flawed tests or descriptions, a share of that hard subset only (OpenAI, first-party, observed 2026-10-28). Its estimate that about 30% of SWE-bench Pro tasks are broken is one lab's view, and no independent replication was found (OpenAI, observed 2026-10-28). Cost per merged PR has no good dataset either: the best public attempt is one practitioner on one feature, where a review gate merged code passing as few as 12 of 38 oracle tests (Insight APAC practitioner blog, single feature, observed 2026-10-28). Treat metric 6 as an internal trend line, never a benchmark. Small lanes also swing: that is why the board has a minimum sample and a hold exit.

Key Takeaways

• Pair every throughput number with incidents per PR, unreviewed-merge share and revert rate on the same row.
• Write a forecast before the lane starts, because felt speed is not measured speed.
• Name the denominator of every share, and drop any AI-code percentage that lacks one.
• Build a private eval from your own reverts, because public benchmarks decay.
• Count what came back, not who typed it.
• Set tripwires against baseline bands and minimum samples, then let them pause the lane.

What You Have Now

factory/scoreboard.md - six metrics with definition, denominator, guardrail pairing and the tripwire that pauses a lane. Keep it in the factory/ directory beside release-path.md from lesson 10, whose rollback owner supplies the incident numbers, identity-ledger.md from lesson 12, which says which PRs are agent PRs, and context-map.md from lesson 13, which lets you split any metric by service tier. It serves the Prove stage: the file that says what the line is allowed to keep doing, and the numbers that stop it. The capstone project folds it into the factory charter.

Further Reading

• Faros, "The AI acceleration whiplash" - the two-sided throughput and incident telemetry, with the vendor's own caveats on causality.
• Faros, "The AI code quality mirage" - why a ratio can stabilise while the absolute count of incidents keeps rising.
• Google Cloud, "Announcing the 2025 DORA report" - the survey evidence for AI as an amplifier of throughput and of instability.
• METR, early-2025 study of AI and experienced developers - the randomised design behind the perception gap, with its own limits stated.
• OpenAI, "Separating signal from noise in coding evaluations" - a lab auditing a public benchmark, and why a private set is safer.
• Uber, "Running a software factory efficiently" - how one company names outcome metrics per agent and says what its share figure does not mean.

Exercises

1. Compute your last month by hand. For one repo, count merged PRs, merges with no approval and no review gate, reverts within fourteen days and incidents traced to a merged PR. Print each as a rate with its denominator written next to it.
2. Write a forecast and four tripwires. Pick one lane you are about to switch on. Date a forecast of throughput and incidents per 100 merged PRs, then write tripwires for metrics 2, 3 and 4 against your own baseline bands.
3. Audit a published claim. Find one "percent of code written by AI" figure from a company or vendor. Name its numerator, its denominator and the outcome printed beside it, then rewrite the claim as the row you would accept.

Check your understanding