Making a Flaky Test Suite Trustworthy Again
Measure flakiness with evidence, fix the confirmed offenders by root cause, and guard against their return.
Gantry's safety rests on one thing: it runs the test suite itself and treats a green result as proof that a change is done. That only works if green is trustworthy. A flaky test — one that fails intermittently under concurrency or timing pressure, not because the code is wrong — breaks the deal in both directions. A spurious failure can stop a good run, and the noise it adds can bury a real regression in the crowd.
This job set out to make Gantry's own Rust test suite trustworthy again, on an evidence-first footing. It built a repeatable command that reruns suspect tests under stress to force unstable failures into the open, a report that ranks offenders by measured failure rate, and durable snapshots so a later run can tell a genuine fix apart from a test that merely happened not to fail this time. With that evidence in hand it traced the confirmed offenders to their causes and repaired them — removing the races rather than papering over them with retries or longer timeouts — and left behind guardrails and a contributor guide so new flakes are caught before they reach the default test run.
Build
The plan cut the work as measure, then fix, then prevent, and forbade any fix before evidence for it existed. The measurement chain held in dependency order; the boundary between fixing a flake and preventing its return did not, and a review redrew it mid-run.
The load-bearing decision was to make evidence a precondition for every edit. Nothing could be repaired until it could be reproduced and ranked, so the run built its instruments in dependency order: a deterministic stress runner that reruns chosen test tiers under controlled concurrency, then a stable report format over the runner's raw output, then an initial ranked baseline of offenders. Each piece could be built and tested against fake runner output before the next leaned on it, and a fresh agent could carry any one of them without holding the others in context. The briefs are explicit that this ordering exists so later fix sprints can compare before and after without rereading ad hoc terminal logs.
The measurement cut came under stress at the baseline. The sprint meant to establish a full-suite ranking could only capture smoke-level full-suite evidence, and its own plan-impact note said so: only the driver_sprint re-enumeration class was confirmed by repeated focused samples across both synced and non-synced checkouts. That kept the remediation honest — the fix milestone targeted the one offender the evidence actually supported rather than the whole ranked offender list.
The boundary that did not hold was the split between the fix milestone and the prevention milestone. Collecting post-fix proof surfaced a residual duplicate-execution failure that the first repair had not fully locked, and the review responded by redrawing the milestone: it re-planned the remediation work, adding several more sprints and pulling the fixture standardization, the antipattern guardrail, and the reliability documentation forward — the exact deliverables the final milestone had been written to produce. The ledger records the consequence plainly: the closing milestone had to reconsider its scope because the middle one already built what it was written to build. What this reveals is that fixing and preventing were not separable once repairing the confirmed flake produced a reusable isolation pattern; verifying that the repair held is what demanded the guardrail, so prevention migrated to where the evidence already was. The closing milestone became an audit that the guardrails still matched the guidance, plus one freshly reproduced residual — an overlay recording-readiness flake — fixed by root cause.
Feature
Gantry's suite once failed intermittently with no dependable way to reproduce a flake or separate noise from a genuine regression. The reproduction command, the ranked report, the root-caused repairs, the antipattern guardrail, and the reliability guide are all still in the tree at today's HEAD.
Before this job there was no repeatable way to make an intermittent failure show itself on demand, and no shared vocabulary for saying whether a green run meant a flake was fixed or merely quiet. The design answers both with a small set of parts that still stand in the tree.
The measurement side is `bin/stress` and its report generator under `scripts/analyze/`, which rerun selected test tiers under chosen concurrency and turn the raw results into a dated, non-clobbering snapshot ranked by failure rate. The snapshots themselves live under the analysis documentation area and are present at HEAD. The remediation side removed real races: the driver_sprint re-enumeration flake was traced to shared stub executables leaking across same-tag fixtures and fixed by isolating them behind a unique-per-fixture temp-name helper, now the canonical safe pattern in the build-engine test support and covered by a parallel-safety test; and the overlay compositor-fallback recording path was fixed so it reliably reaches its cell-size readiness probe instead of occasionally dying while measuring the grid.
The prevention side is a meta-test guardrail that runs in the normal verification path and rejects the known antipatterns — unguarded process-global mutation, readiness-by-sleep, brittle process-id-only scratch names, and the same-tag shared-stub fixture shape — alongside a contributor reliability guide that names the accepted helpers, the rejected patterns, the gate commands, and where the snapshots live. Standing in the live worktree at HEAD, all of it is present and coherent: the guardrail still scans source text for its rules, the reliability guide still documents the verification tiers and the safe helpers, and nearly every file the job introduced survives. Nothing later folded this into a larger design; it remains the suite's reliability workflow.