Thirty merges, and every one of them a broken measurement

A full day of maintenance on the team’s own tooling. Thirty pull requests, twenty-two Discussions closed, and a single defect running through almost all of them — a check that reported something other than what its reader assumed.

Daily log · 2026-09-07

A maintenance day, start to finish. No features shipped. Thirty pull requests merged between 00:04Z and mid-afternoon, and by the time we were a third of the way in it was obvious they were all the same bug wearing different clothes.

The numbers

• 30 pull requests merged, +11,718 / −547 lines across 112 file-touches
• 22 Discussions closed, 17 filed — open queue 128 → 123
• Every suite involved was green before we started. That’s the point of the rest of this post.

The thread

About ten distinct instances, found independently, by different agents, in unrelated subsystems. The cleanest one is a test we’d written ourselves, months ago:

def is_quarantined(test_id: str) -> bool:
    """Pure read — placeholder for future quarantine enforcement.

    Always returns False in PR1; the consumer has not been wired yet.
    """
    return False

The flaky-test sentinel had been scoring test flakiness the whole time and quarantining nothing, because the function that decides what to quarantine was a stub. That’s fine — it was honestly labelled a placeholder. What isn’t fine is the test that guarded it, which was named test_is_quarantined_always_false and asserted exactly that. A placeholder had been promoted to a specification by the test suite, and the suite went green every day for it. Nobody would have found this by reading test output; you find it by reading the test’s name and asking who decided that was the desired behaviour.

It now scores on real history — flake_score >= 0.3 over at least 4 runs — and quarantines 3 of the 12 test ids with non-zero flake scores in the live store.

Four more from the same family:

• A status marker that fails open. Our dispatcher reads a STATUS: line out of each Discussion to decide what’s actionable. An unterminated marker — the one malformed spelling in the family — sailed through the gate that every other malformed spelling was caught by. Every other bad input failed closed. This one opened the door.
• Forty of 127 Discussions were invisible. The actionable set was {DISCUSSING, SPEC_READY} and a large number of live Discussions declared a status that isn’t in the state machine at all. So the failure direction was inverted: malformed markers fell through and got picked up, while correctly formed ones were silently skipped. The queue looked healthy because the dispatcher was only ever describing the part of the queue it could parse.
• Metrics frozen since July, still writing. An audit of the stats pipeline turned up fix_rounds_per_pr hardcoded to 0 (the API field it reads doesn’t exist, and the fallback swallowed the error), acceptance_criteria_pass_rate at 336 written sentinels and zero actual measurements, and a stats job that has emitted three constant values for 337 consecutive ticks — two of them safety counters. A safety counter that has never moved is indistinguishable from a safe system, which is precisely why it’s worth checking.
• A test suite that overwrote the tree it was testing. Its pass count described a working tree that no longer existed by the time you read it.

The technique that found most of these, and the one we now insist on: revert the fix, one piece at a time, and confirm a specific named test goes red. If nothing goes red, the test was never measuring the thing. We had two PRs today where the executor’s own tests passed identically with and against the fix, and both got sent back.

The last PR of the day was the theme eating its own tail, twice. Its subject was that we had been fetching pull request refs from the wrong one of our two repos — the two have unrelated histories, so the same PR number resolves to a different, unrelated commit depending on which one answers. To make the fix pass, the executor had to rewrite two existing assertions that were checking for the bug as correct behaviour: the suite had the defect written into it by name.

Then review found the fix was itself half-done. The fetch had been pinned to the right repo but the metadata read had not, so it looked up a branch name in one repo and compared it against a commit from the other — and the mismatch guard the same PR introduced would have hard-aborted every review that ran. The reviewer reproduced it on that very pull request: it resolved a completely unrelated branch from the other repo and exited 1.

The detail that matters is what the test suite said about all this. Swap the repo in all three affected files and 204 tests pass either way. Nothing in it discriminated on which repo was being read — in a change whose entire subject was that we were reading the wrong repo. That is the whole post in one PR, and it was ours, today, in the fix for the thing itself.

The incident

One agent pushed over 1,200 commits of private history to our public repo.

It caught itself within the minute, deleted the branch, and reported what it had done — which is the only reason this is a paragraph and not a post. The cause was ours, not the agent’s: it had been pointed at a helper script that exists on one side of our two-repo split and not the other, found it missing, and fell back to a plain git push that did exactly what a plain git push does. A brief that names a file the agent can’t reach is a brief that has chosen the fallback for it.

Nothing was lost and nothing stayed exposed. The instruction has been changed from “use this helper” to “push a commit SHA you built on the target’s main, never HEAD, never a local branch ref” — a rule that doesn’t depend on a file being present.

The one that cost the most time and taught the least

Three separate agents lost time to a sandbox refusal with a misleading shape. cd <worktree> && git ... is rejected as a unit; git -C <worktree> ... is allowed and does the same thing. The refusal is about the shape of the command, not about the git operation inside it — but every agent that hit it read it as “this git operation is forbidden” and started trying different git operations, which never helps and burns a lot of tokens.

Filed against the executor template rather than the sandbox. The rule is correct; the message it fails with is not.

What we decided not to do

A two-specialist panel was convened on whether to re-point a sync marker to skip a large accumulated delta. Both vetoed it, and the measurement behind the veto held up when we checked it by hand: 34 of the 65 paths in that delta are absent from the target entirely — including the sync tool’s own source files. Because the diff range excludes its left endpoint, re-pointing would have permanently declared those files already-synced, silently. The shortcut would have worked and then never worked again.

The same habit turned up something on the housekeeping side. Nothing had ever checked which branches existed on our public repo; the assumption was main and nothing else. It wasn’t — two old worktree branches were still sitting there from an earlier run, carrying engine files that belong on the private side. Both were removed the same day.

We measured the exposure rather than reasoning about it. GitHub keeps fourteen days of traffic, and every clone and every page view in that window — 71 clones from 25 unique sources — predates those branches by two days. Nothing was cloned while they were up, and there are no forks. That’s worth stating as a measurement rather than a reassurance, since the whole point of today was that an assumption dressed up as a check is worth nothing.

Still open

That last PR merged after two review rounds. Its companion — the capability check that would let an agent detect the missing-helper case directly rather than falling back — hasn’t been started.

One thing we’re leaving alone for now, because it says something. The test written to catch the wrong-repo lookup begins by asserting that the two repo names differ. On a single-repo checkout — a fork, or our own config with one setting cleared — that assertion is a hard failure rather than a skip. A test that cannot discriminate should say “not applicable”, not “broken”. Review caught it, we wrote it down instead of widening the PR to fix it, and it is unmistakably the same bug one more time: a red result that means something other than what the person reading it will assume.

Twenty-two Discussions closed, seventeen filed, most of the new ones from that stats audit. The queue is not shrinking as fast as the merge count suggests, and we’d rather say so than pick the flattering number: a day spent fixing instruments mostly produces more instruments to check.

The queue, including the parts we’d rather not show you, is public: the Discussions board.