Merge to Staging, and the Shipper Owns the Rollback

In the year to August 2026, Spotify's engineers went from roughly 8,100 merged changes a month to roughly 17,000 (self-reported by Spotify in a quality review, Spotify, observed 2026-10-28). Spotify describes its classic fleet changes as mostly auto-merged after safety checks, and says agentic changes use the same path. One of the automated changes, an automated dependency upgrade, passed every check the pipeline ran and then broke something for real users in production. Spotify's own reading was that the volume of change had grown faster than its review, testing, rollout, observability and rollback controls could adapt. It also reported that it had found no reviewed incident where AI-authored code was a material direct cause, though it called the method informal. So the code was not the story. Every check had said yes at the moment of the merge, and after the merge nobody in the process was assigned to watch what the change did next. Decide now: in your line, who is watching the hour after the merge button is pressed, and what are they allowed to do about a red graph?

This lesson gives you factory/release-path.md - merge target, staging validation, canary checks with false-positive tracking, rollback owner and the human release step. It serves the Run stage of the loop: the part of the line that keeps moving after a gate says yes, and decides how a bad change gets back out.

What is actually going on

A merge answers one question: may this change join the shared code? It does not answer what the change does to the combined state of the branch, to a customer using the product, or to production under live traffic. Those are three separate questions, and each has a place in the line. A line that skips one has not removed the question. It has chosen to find the answer later, in front of users.

The release path is the chain that answers them in order. Merged work lands on a branch that deploys to staging, not to production. A merge queue sits in front of that branch. It builds a temporary branch holding the target plus the queued changes and re-runs the required checks on that combined state, so a change that was green alone cannot turn the shared branch red (GitHub Docs, observed 2026-10-28). Uber's SubmitQueue paper is the research version of the same idea (Uber, EuroSys 2019, observed 2026-10-28). A validator then uses staging the way a customer would, and every defect it finds goes back into the line as a new issue instead of a patch to the release. Only after that does a named human cut the production release, and the release rolls out in phases with a check at each boundary that can pause or roll it back.

The deploy itself stays deterministic. Press accounts of Amazon's Stores organisation (secondhand) describe deployment as ordinary AWS tooling, with a human approving every mutating AI step (The Register, observed 2026-10-28). The Claude Code team describes the same shape in its own process, self-reported: ship behind a flag to employees first, put a build on a latest channel before stable, keep a weekly on-call, and switch a flag off without a new deploy (Anthropic Claude Code team, observed 2026-10-28).

Read the diagram for two things: the direction defects travel, and which arrow the human release step sits on.

The issue arrow goes back to the author's queue, not into the release branch, so the release stays a known batch. And the person who cuts the release is the same person who stays on the phase checks. That person is the shipper. A rollback nobody owns is a rollback nobody runs.

The rule and its condition

Use a staging-first path with a human-cut production release when a change can reach real users and reverting it costs more than one command. Use phased canary checks, with automatic pause or rollback, whenever exposure is gradual. Otherwise, for a docs-only lane or a flagged change with a one-command revert, you may merge straight through, but you still name the rollback owner before the merge, not after the alert.

The condition is observable. Open your last ten merged changes and ask two questions of each: could it reach a user, and what does the revert cost? If the answers vary, you need the lanes from lesson 7 to decide which path each change takes. The medium lane there says merge queue, staged, auto-rollback, human sees a summary. This lesson is what those four words mean in a file.

Three ownership rules complete the rule. The shipper, the author or the person who sponsors the lane, watches the deploy even after an AI reviewer has approved the change. Approval is not attention. A human cuts the production release, and a bot approval never fills that slot. And every canary check carries a record of how often it was wrong, because a gate that cries wolf gets muted, and a muted gate is a missing gate.

Sources for the table's pattern: GitHub Docs (queue), Meta's rollout report and Spotify's review, both linked in this lesson.

Worked case

Illustrative composite. Dana Okafor leads payments engineering at a fictional company that runs an invoices-api service in three regions. Everything below about her service is invented to show the shape. The Meta figures are real and linked.

Last quarter an agent-authored dependency bump passed CI and the AI reviewer, deployed to all regions at once, and raised payment failures for several minutes before anyone noticed. The alert posted to a shared channel and the author of record was an agent. Dana has to write factory/release-path.md. One engineer cuts releases. The smallest region has thin traffic. The existing alert, error rate above a fixed ceiling, fires so often the team muted it. She has three designs.

Design A: keep the one global error-rate alert and wire it to automatic rollback. Design B: a different check per rollout phase, a control group where one exists, a static ceiling where none does, and a true or false label on every verdict, with noisy checks demoted. Design C: skip the canary and rely on a long staging soak plus the human cut.

Decide now: which design, and what happens to Design A on her team within a month?

Design B. Design A fails in two predictable ways. Phase one has two hosts, too little traffic for a fixed ceiling to mean anything, and the last phase has no control group left to compare against, so one threshold is wrong somewhere in every rollout. Worse, the alert is already muted. Connecting a check people ignore to a rollback button turns an annoyance into rollbacks of good releases, and the team's answer will be to mute it again. Meta's authors describe exactly this from their own fleet: the main cost of a rollout gate is false positives, owners stop trusting checks that cry wolf, and most owners never touch the default check (self-reported by Meta, Meta, observed 2026-10-28).

Design C fails on a different edge. Staging carries no real customers, and Spotify's upgrade passed every check before it hurt real users in production. A soak finds what staging reproduces. It does not find what only production traffic exercises.

Design B is also what Meta built. It runs a check per tier and phase, matches the check to what the phase can support (relative to a control group, or static), keeps predicates so a check does not fire where there is no control group or too little power, and labels every verdict true or false so owners see a catalog of their low-quality checks. After a year of that program, Meta reports its fleet-wide check false-positive rate falling from 12.1% to 2.7%, checked by two senior engineers who classified 500 random failures and found the under-counting small, about 0.5% (Meta, observed 2026-10-28). This is a self-reported experience report about ordinary service rollouts at Meta's scale. It says nothing about AI-authored changes, so it teaches the gate-quality mechanism and not a result for your line.

Here is the file Dana writes.

Each section answers one of the three separate questions from earlier. Section 1 covers the combined state, section 2 the customer's view, sections 4 to 6 production exposure, and section 3 holds the single human decision between them. The revert count in section 6 is the quality signal Uber says it tracks for each managed agent, alongside outcome metrics, though it does not publish the values (Uber, observed 2026-10-28).

The demotion rule in section 5 is the part teams skip, so draw it once as a loop.

The loop closes on measurement, not on the owner's say-so, so a demoted check cannot drift back into blocking on hope.

Illustrative composite. A quarter of Dana's invoices-api log might read: p95 latency check fired 10 times, 4 labelled FALSE, so it is over budget and demoted; payment-success check fired 3 times, 0 FALSE, stays blocking. The numbers are invented. The shape, a label on every firing and a budget that demotes, is the point.

The rejected designs cost her an outage she already had once. Design A costs another one the first time the thin-traffic check pages at 3 a.m. and the team learns to ignore the page.

Faded case

Skip to the open case if this is familiar. A fictional mobile-API team ships a release in three phases. The release candidate takes mirrored traffic only. Then it goes to one region, then to every region. The shadow phase has a control build running the old code, and the one-region phase can compare against the other regions. The global phase has no control left, because every region runs the new build. Their draft has three checks: error rate against a control, p95 latency against a control, and a static CPU ceiling.

The CPU ceiling is the team's oldest check. In the last month it fired four times, and the rollback owner's notes say three of those pauses were caused by a batch job and not by the release. Nobody has labelled them formally, and the check is still blocking. The team lead wants to know three things before the release template is signed: where each draft check is allowed to fire, what happens to the CPU check, and who writes the true or false label on each verdict and by when.

Use the worked case as your pattern. The only new decision is the one the CPU check forces: a check that is valid in a phase can still be too noisy to block there.

Open case

A small team's agent line merges dependency bumps to main, and main deploys to every region the moment CI is green. An AI reviewer approves, the human-approval slot is filled by that same bot, and the author of record is an agent. Nobody on the team has ever been named as the person who watches a deploy. At 02:10 on a Sunday the checkout error rate doubles. The alert posts to a shared channel that nobody is assigned to read, and the one engineer who notices at 08:00 does not know whether she is allowed to roll back, because the repo has no written rollback owner.

Write the release path this team lacks in five lines or fewer: the merge target, how the change is validated before customers see it, who cuts the release, what the first phase checks, and who owns the way back. The quiz grades your answer.

Where this breaks

No public data exists for agent-authored changes. The numbers behind the checks come from elsewhere. Meta's false-positive figures are about ordinary service rollouts and explicitly not about AI changes. Spotify says it found no AI-authored code as a direct cause in reviewed incidents, by an informal method. The Claude Code team says it has no circuit breakers yet that tighten approval when revert or incident rates move, and its engineers want them before more zero-human merges. Treat this lesson as the mechanism teams are told to build, not as a measured result for agents.

Staging only reproduces what staging contains. The queue checks what CI covers and nothing semantic. A validator walking journeys finds what its journeys touch. Spotify's failed upgrade reached real users after passing checks, which is the gap staging cannot close alone. Canary phases exist because of that gap, and they need enough traffic and a control group to say anything.

False-positive tracking costs people. Meta validated its labels with two senior engineers reading 500 failures (Meta, observed 2026-10-28). A small team has fewer verdicts and fewer reviewers, so the budget is noisier and the labelling load lands on one shipper. And a human release step can habituate into a stamp, the failure lesson 7 describes. Guard it with the revert count, not with trust.

Key Takeaways

• Land merged work on a staging branch, never straight on production, whenever a change can reach users.
• Send every staging defect back into the line as a new issue, never a patch on the release.
• Match each canary check to what its phase can measure, and let relative checks sit out phases with no control group.
• Label every pause or rollback true or false, and demote any check that goes over its false-positive budget.
• Name the shipper before the merge, and keep a human, not a bot approval, on the production cut.
• A rollback nobody owns is a rollback nobody runs.

What You Have Now

factory/release-path.md - merge target, staging validation, canary checks with false-positive tracking, rollback owner and the human release step. It lives in your factory/ directory beside the risk lanes from lesson 7 and the blueprint from lesson 8, and it supplies the medium lane's staged, auto-rollback promise as named files, phases and owners. The readiness score from lesson 9 says the line may start. This file says how a change leaves it and who brings it back. It completes the Run stage of the loop and feeds the incident record you will read in Prove.

Further Reading

• Spotify, "AI changed how Spotify builds: what we learned and fixed about quality at higher velocity" - the postmortem behind the opening, including the two questions Spotify added to its incident retros.
• Meta, service health checker experience report - phase-matched checks and the one-year false-positive program, with the labelling method.
• Meta, Conveyor deployment system (OSDI 2023) - the companion paper on the deployment system the health checks gate.
• GitHub Docs, managing a merge queue - how a queue tests the combined state and which checks it requires.
• Uber, SubmitQueue (EuroSys 2019) - the research behind testing the merge result and its throughput trade-offs.
• Anthropic Claude Code team, code review process - flags, a latest channel before stable, on-call and the admitted gaps.

Exercises

1. Rewrite one rollout. Take your last production release and fill sections 1 to 3 of the template: merge target, the journeys a customer would walk, and the one named human who cuts it.
2. Draw your phases. List every rollout phase you have, and for each say whether a control group exists. Write one check per phase and mark the phases where a relative check would be meaningless.
3. Audit a gate that cried wolf. Find one alert or check your team mutes. Estimate how often it fired and how often it was wrong from the last month of history, set a budget, and write the demotion and re-promotion rule for it.

Check your understanding