A Holdout Scenario Must Assert Exact Figures

Illustrative composite. A small team ran an unattended line against an internal expenses service. The issue asked for a monthly report of what each employee is owed. A hidden acceptance scenario guarded the merge, and it asserted one thing: after the run, every ledger balance sums to zero. To test the test, someone planted a defect in a copy of the service. The report returned only the first approved claim for each employee. The scenario stayed green, and it could not have done anything else. Every ledger entry is half of a balanced pair, so any subset of entries still sums to zero, and so does an empty list. In the real service that defect would have underpaid every employee with more than one claim, finance would have learned about it from complaints, and someone would have repaired the month by hand. The scenarios were hidden from the builder, written by another person, and run by a validator that never saw the plan. Every safeguard worked except the one that mattered, the assertion itself. Decide now: what must a hidden scenario assert so that a build returning the wrong rows cannot pass it?

This lesson gives you factory/holdout-set.md - 3 hidden scenarios with hand-computed expected figures, where they are stored, and the red control for each. It serves the Gate stage of the course loop (Slice, Gate, Run, Wrap, Prove): the acceptance check that decides whether a pull request from an unattended run may reach the merge decision.

What is actually going on

The failure has a name worth keeping: a property check. A property is a statement that stays true across many different outputs, wrong ones included. "Balances sum to zero" is true for the full ledger, for half of it and for an empty one. "The report is not empty" is true for a report with a single row. Properties catch crashes and corrupted entries, and they are cheap to write, which is why builders and test generators reach for them. What they cannot do is separate the right answer from the nearly right answer, and agents produce nearly right answers all day.

An exact figure can. If the October fixture holds known claims for three employees, only one report is correct, and a person can write down its totals with a calculator before the build exists. A build that returns a subset, counts a claim twice or drops a boundary case produces a different number and fails. A property accepts the right answer and many wrong ones. A figure accepts one.

Exact figures only matter inside a real holdout, and a holdout needs three separations. The first is storage: the scenarios sit where the builder can neither read nor edit them. StrongDM describes keeping end-to-end scenarios outside the codebase, the way a machine-learning team keeps a holdout set, so coding agents cannot see or game them (StrongDM, observed 2026-10-28). The reason is practical. Tests inside the repository can be rewritten to match the code, and StrongDM reports that early agents wrote code that simply returned true to satisfy narrow tests (self-reported by StrongDM, echoed by Stanford CodeX, observed 2026-10-28). The second is role: someone other than the builder writes the scenarios, and a validator reads the issue and the running build, never the plan, and returns a verdict. A validator that reads the plan inherits the builder's misreading of the issue. The third is a thin verdict: scenario ids and pass or fail, never the expected figures, because every leaked figure lets the builder fit its output to the key.

One honest limit on the evidence: StrongDM publishes its mechanics but no defect rates, so nobody outside can say how often such scenarios catch what a reviewer would have caught.

The diagram shows who touches what. Look for the arrows into the store.

The builder has an arrow to the gate and none to the store. The validator holds the only read arrow into the store, and the only thing that travels back toward the builder is a list of failed ids with the issue clause each one tests.

The rule and its condition

Use hidden exact-figure scenarios as the acceptance check when a pull request can reach merge without a human reading the diff and the feature's output can be computed by hand from a fixture. Otherwise, when the output is judged by taste or a human reads every change, use review, and keep scenarios as a second opinion.

The condition is observable on paper: can you write the expected number from the fixture without running the code? If you can, you have a figure. If all you can write is "looks right", you have an opinion, and an opinion needs a reader, not a holdout.

Four parts make the rule, and each one closes a different escape:

1. Store the scenarios where the builder holds no read or write grant.
2. Let the validator judge the outcome against the issue, with the plan and the builder's summary kept out.
3. Assert hand-computed figures, not properties that hold for subsets.
4. Prove each check can fail by running it against a deliberately broken build, the red control, before you admit it.

If the broken build can pass it, it is a property, not a holdout.

Keep your properties as a cheap net for corrupted data. They do not count toward the holdout set, because they cannot reject an answer that is wrong but internally consistent.

The red control is the part teams skip, so it gets its own picture. Every candidate scenario has to survive two runs before it earns a place in the set.

There are only two ways in and one way out. A scenario that cannot fail its red control never reaches the set, however sensible it reads.

Worked case

Illustrative composite. A four-person finance-tools team runs an unattended line for an internal expenses service. The issue the builder receives says: the report lists, per employee and month, the sum of approved claims; pending and rejected claims do not count; meals are reimbursed up to 25.00 per employee per day and the excess is flagged; months follow UTC. The draft holdout has three checks: the ledger sums to zero, every employee with an approved claim appears in the report, and a judge model reads the report and says whether it looks right. All figures in this case are modeled, not measured.

Decide now: the draft goes live tonight. Which change do you make first?

• A. Add more properties: totals are non-negative, no claim is counted twice, months do not overlap.
• B. Replace the draft with three scenarios whose figures you compute by hand, store them outside the service repository and give each a mutant it must fail.
• C. Let the validator read the builder's plan so it understands what the report is meant to do.

The answer is B. Start by seeing why the draft fails. The first-record mutant passes the zero-sum check. It passes "every employee appears" too, because each employee still has a row. A judge model may well pass it, since a short report looks tidy. Option A is the tempting repair. Each new property is cheap, and each is satisfied by some wrong build: "no claim is counted twice" is happy with a build that drops claims. Option C feels like helpfulness and does the opposite. If the builder's plan read the meal cap as per claim, a validator that reads the plan expects the same misreading. Independence is what you are paying for, and C spends it.

Option B takes six steps:

1. List the behaviours a wrong build would plausibly get wrong while staying consistent: grouping, the daily cap, the month boundary.
2. Write a small fixture for each, in its own empty database.
3. Compute the expected figures by hand, from the fixture only.
4. Write one mutant per scenario that breaks exactly that behaviour.
5. Put fixtures, figures and mutants in a store the builder's token cannot reach.
6. Decide what the verdict reveals: ids and issue clauses, no figures.

The table shows what each scenario expects, what its mutant returns, and the property that mutant still satisfies.

Illustrative composite. Every figure in the table is modeled for this lesson and computed by hand from the fixtures in the file below; none is measured. Check one yourself: Dana's three approved claims are 48.20, 112.75 and 19.35, and they sum to 180.30. The rejected 300.00 flight and the pending 60.00 taxi do not count.

Read the right-hand column twice. Each mutant satisfies a tidy, believable property. That is how property-only holdouts get written: every assertion is true, and none of them is selective. The cost of the rejected options is a green gate that releases wrong totals. The cost of B is the scenario author's time, paid once per feature, plus a dated edit whenever a rule changes. That cost is the point: someone has to know the right answer before the build, and the holdout forces them to write it down.

Here is the deliverable in full.

Two details carry weight. The attempt cap matters because each validator run leaks a little: even a bare pass or fail tells a determined builder which of three scenarios it is failing. And the base-commit run in the controls procedure catches a quieter failure. One vendor, GetPullRequest, describes its own agents reporting a task as done with nothing changed, and a green status with nothing behind it (vendor post, self-reported, GetPullRequest, observed 2026-10-28). A scenario that already passes on the untouched base commit would wave that through.

Faded case

Skip to the open case if this is familiar. The same service gets a refund feature, and you are the scenario author. The builder will see only the issue clause below. You hold the fixture, and nobody else may. Your job has three parts, in order. First, work out the expected figures by hand, without running anything, and show the arithmetic the way H1 does. Second, name a red control: a plausible wrong build, not an absurd one, since absurd mutants are caught by almost any check and prove nothing. Third, name one property that the red control still satisfies, because that property is exactly the assertion you must refuse to rely on. Take a minute with a calculator before you open the answer.

Illustrative composite. Issue clause: net reimbursed is the paid amount minus accepted refunds; a refund larger than the remaining balance of its claim is rejected and changes nothing. Fixture, October, own empty database: claim c31 Amir hotel 240.00, paid on 3 Oct; claim c32 Amir train 86.50, paid on 5 Oct; refund r1 of 60.00 against c31 on 10 Oct; refund r2 of 200.00 against c31 on 12 Oct; refund r3 of 86.50 against c32 on 15 Oct. All figures are modeled.

Open case

Illustrative composite. A loyalty service gives points per purchase, and points expire twelve months after issue. A fleet is changing the rule to the end of the twelfth month after issue. The existing hidden check asserts that no balance goes negative and that issued minus expired minus redeemed equals the balance. You own the holdout set. The change merges unattended tonight if the validator says pass.

You have no scaffolding beyond this: no template, no list of mutants, no hint about which date is the dangerous one. Write the scenario you would admit, its fixture, the exact figure it expects and the red control it must fail. Then say in one sentence where you would store it and what the validator would be allowed to tell the builder when it fails. Pick the fixture dates with care, because the whole point of the change is that two readings of the rule disagree on some of them. The quiz grades your choice.

Where this breaks

Holdouts relocate review; they do not remove it. A sceptical commentary on StrongDM argues that holdout scenarios are smart but amount to review moved upstream, and that edge cases are unbounded (The Pragmatic CTO, observed 2026-10-28; opinion, not measurement). It is right. Someone must know the correct figures and write them down, and three scenarios cover three behaviours, not the feature. If you cannot afford to compute figures by hand, you cannot afford the holdout, and a human should read the change.

An AI judge grading AI is circular. Stanford CodeX points out that a builder and an inspector built from similar models share blind spots, and that a satisfaction score invites Goodhart effects (secondhand analysis, Stanford CodeX, observed 2026-10-28). Hand-computed figures help only while a person, or an independent calculation, produced them. If a model writes the scenarios from the same issue the builder read, the independence is gone. For fuzzy outputs you still need a judge, and then a human should audit a sample of its verdicts.

Fixed sets decay. OpenAI stopped reporting SWE-bench Verified after auditing 138 problems and finding 59.4% had flawed tests or descriptions, and after finding that frontier models could reproduce some gold patches (OpenAI, observed 2026-10-28). That audit covered a hard subset only, and OpenAI also consumes these benchmarks. The lesson for you is narrower: a test set that stays fixed and visible gets leaked and gamed, so rotate scenarios and retire any whose figures escape.

Key Takeaways

• Store acceptance scenarios where the builder holds no read or write grant.
• Compute every expected figure by hand from the fixture; never run the code under test to get it.
• If the broken build can pass it, it is a property, not a holdout.
• Run each scenario on a known-good build and on a red control before admitting it.
• Return failed ids and issue clauses to the builder, never expected figures.
• Retire and replace any scenario whose figures leak.

What You Have Now

factory/holdout-set.md - 3 hidden scenarios with hand-computed expected figures, where they are stored, and the red control for each. Keep the master copy in the holdout store, and keep a pointer to it in your factory/ directory beside the review gate from Lesson 4, so that the same pull request meets a fresh-context reviewer whose output is advisory and a validator whose verdict is a required check. This is the Gate stage for acceptance: the check that fails a build returning the wrong rows even when every other safeguard worked. Agent Fleets, lesson 12, introduced the hidden acceptance criterion. Your file now says what that criterion must assert.

Further Reading

• StrongDM, Software Factory: the primary description of scenarios kept outside the codebase and judged by a satisfaction measure.
• Simon Willison on the StrongDM factory: an outside reader's account, including the cost of running that way.
• Stanford CodeX, "Built by agents, tested by agents, trusted by whom?": the circularity and liability critique.
• The Pragmatic CTO, on the software factory: the argument that holdouts move review upstream.
• OpenAI, why SWE-bench Verified is no longer evaluated: a first-party account of a fixed public test set decaying.
• GetPullRequest, why lights-out coding fails: vendor-written concrete failures, including green status with nothing behind it.

Exercises

1. Pick one feature in your repository whose output you can compute by hand. Write one scenario with a fixture and an exact figure, then write down a property that a wrong build would still satisfy.
2. Write the red control for that scenario as a small deliberate change in a throwaway copy. Run the scenario on the known-good build and on the mutant, and record both results.
3. Fill in the full file for three scenarios from your own repository: storage, access, attempt cap, figures with initials and dates, and a red control each. Then test the boundary: try to read the store with the builder's credentials and note what happens.

Check your understanding