skip to content
Replays

Test Suite Redesign

Tests Stop Owning the Machine

The root and web gates move toward fakeable driver logic, bounded processes, explicit tiers, and structural page checks.

Gantry milestones

4 milestones 19 tasks 46 agents

7 plan 22 execute 26 review 1 fix

244k peak context

170k median execute

The root and web gates move toward fakeable driver logic, bounded processes, explicit tiers, and structural page checks.

8h 2m total 14m 39s per task

6m 17s plan 4h 24m execute 1h 41m review

153 edits 1.3k commands

codex harness

This run rebuilt Gantry's own test strategy around the parts of the system that had been making unattended work brittle: driver decisions hidden behind real repositories and shell stubs, process-global state in concurrent tests, hung runners, overloaded root gates, and Rails checks that failed on ordinary copy edits. The decomposition is useful because it does not treat the test suite as one cleanup pass. It first cuts a fakeable driver boundary, then hardens process ownership and timeout behavior, then separates root verification into explicit tiers with stress evidence, and finally applies the same anti-brittleness standard to the website.

The facts show a run that mostly held its planned boundaries, with the real stress concentrated in driver-mode coverage and in milestone notes saying the driver seam was still incomplete after early passes. Later milestones consumed those notes instead of pretending the first boundary was done. In the current tree, the surviving feature is a practical testing contract: ordinary checks stay local and structural, slow or process-owning work is named, and the remaining exact-copy web assertions live outside the default gate.

How this walkthrough is structured

Feature

What did the run build and ship?

Gantry now has fake-backed driver tests, bounded gate and runner behavior, explicit root tiers, stress evidence tooling, and structural web assertions still visible in the tree.

Build

How did Gantry structure the work?

The build moved from driver isolation through process hardening into tiered root and Rails checks, with stress recorded where real driver coverage resisted the first cut.

Milestone 1

Driver Logic Seam

5 tasks 20 agents 4h 3m wall time 244k peak context

The current tree still has driver tests and a world boundary around goal startup, but later refactors moved much of the original build-driver layout under the run engine. What survives most clearly is the testing posture: decision coverage lives in in-process scaffolds where possible, while real repository and subprocess behavior is treated as integration surface. The early milestone's incomplete note also survives as context for why later tiering audited the driver tests again.

This milestone carried the riskiest architectural split: separating driver decisions from the real process world they had been exercising directly. The nested sprints kept the work small by adding the adapter first, then migrating sprint, failure, mode, and goal coverage onto it. Most of that sequence passed at its boundaries, but the mode-driver sprint hit the only recorded red gate and needed repair. The milestone closed green while recording that the real integration surface was not yet as small as the plan wanted.

Driver World Seam

2 agents 40m 9s wall time 244k peak context

The run-era module names have since shifted, but the current run engine still keeps driver support behind world-style traits and shared driver services. Tests can construct scaffolds and exercise control decisions without launching the full external path for every case. The real adapter remains the route for command-line behavior, so the feature is an internal test seam rather than a user-facing command.

This sprint established the first fakeable boundary before any broad migration depended on it. Its brief asked for production behavior to keep using the real adapter while a representative decision could be proven without real git or spawned agents. The scope was narrow enough for a fresh agent to hold because it was a seam and a proof case, not a suite rewrite. The run facts show a clean boundary for that foundation.

Sprint Flow Coverage

4 agents 44m 36s wall time 212k peak context

Sprint-flow behavior remains covered in the run-engine tests through shared scaffolds rather than only through shell-backed end-to-end fixtures. The current files have been renamed around the run module, but their helper layer still models plans, gates, ledgers, and outcomes in process. That leaves the production sprint path protected while reducing dependence on real repository setup for ordinary decision assertions.

This sprint consumed the new seam by moving sprint-flow assertions onto fake-backed control paths. The brief kept the agent focused on ordering, resume, gate, review, and finalization decisions in the sprint driver. It ran more than a single pass through the unit pipeline but stayed green, which points to coverage expansion rather than a broken boundary. The important cut was that sprint behavior could now be asserted without restaging the whole outside world.

Failure Accounting Coverage

4 agents 46m 20s wall time 174k peak context

Failure classification is still a first-class concern in the run tests and gate machinery. The current tree keeps tests around driver-loop failure paths, post-revert infrastructure outcomes, and gate result handling. Some original fake-coverage lines were later rewritten, but the behavior remains represented as control logic rather than only as slow external simulations.

This sprint narrowed the problem to how driver failures are counted and reported after failed gates, malformed output, and infrastructure-style exits. That was a good memoryless-agent unit because it could lean on the fake world from the previous sprints and focus on classification rather than setup mechanics. The facts show a green pass through the sprint boundary. The value of the cut was making failure accounting direct enough to test without reproducing every external failure in a real process.

Mode Driver Coverage

5 agents 1h 11m wall time 224k peak context 1 fix

Mode-driver coverage is now partly absorbed into the generic run and pattern tests that replaced some older driver modules. The current tree still has driver tests for loop, map, milestone, sprint, and goal routes, with shared fixtures making their setup less dependent on external state. The exact sprint contribution is no longer isolated cleanly because later pattern work changed the surrounding architecture.

This sprint expanded the seam across milestone, loop, map, and routing behavior, which made it the widest cut in the milestone. The gate went red here and a fix pass followed, locating the difficulty in applying the fake-backed model across driver modes rather than in the seam itself. Review also recorded a note, so the boundary produced useful feedback instead of just a green commit. The sprint still completed without forcing a run-level recovery.

Goal Coverage and Real Integration Surface

2 agents 33m 33s wall time 213k peak context

Goal startup now has an explicit world trait in the run engine, and goal tests still exercise baseline, resume, runtime, and finish behavior through that boundary. The real integration surface has since been reshaped by later work, but the distinction remains visible in tests and docs. The survivor is not every original file, but the fact that goal decisions can be tested without treating the whole command-line path as the only oracle.

This sprint closed the milestone by adding goal-driver coverage and drawing the remaining real integration surface. Its brief depended on the prior fake tests but also required an honest list of what still needed real subprocess and repository behavior. It passed green, while the milestone ledger still reported that the wider driver-test reduction was incomplete. That made the next milestone inherit a concrete triage problem rather than a vague cleanup request.

Milestone 2

Process Isolation

5 tasks 13 agents 1h 32m wall time 212k peak context

The current tree keeps the process-hardening results in configuration resolution, gate execution, nextest profiles, the root test script, reliability docs, and process-aware test helpers. Ordinary tests no longer need to serialize around every environment or current-directory mutation, and hung fallback tiers now have terminal verdicts. Later refactors moved module names, but the operational contract remains part of the root gate.

This milestone picked up the incomplete driver-test triage while also removing shared process state as a suite-wide bottleneck. Its sprints form a clean chain: classify remaining real driver coverage, pass process context explicitly, narrow process ownership, bound runner hangs, and supervise baseline gates. The run facts show this chain stayed green, which suggests the contracts were sequenced well after the first milestone exposed the unresolved surface. The milestone still recorded that some real loop decision tests needed later audit.

Driver Triage

2 agents 19m 22s wall time 212k peak context

The current driver tests retain comments and fixture organization that distinguish production wiring from pure decision logic. Some of the triaged files were later renamed under the run engine, so this sprint's exact line ownership is diffuse. Its durable feature is the explicit expectation that real driver tests need a reason, not just historical inertia.

This sprint was deliberately diagnostic before more timeout and lock work landed. It had to classify the remaining driver integration surface left by the prior milestone and protect coverage while moving more decisions toward fake-backed tests. That bounded task fit a fresh agent because it was an audit and migration pass over known driver tests. It passed green and gave later tiering a clearer, though still incomplete, map of what real integration meant.

Process Context

2 agents 16m 32s wall time 172k peak context

Configuration and overlay-player resolution still expose injectable paths in the current engine. Tests can supply resolved context instead of mutating global environment or changing the process directory for ordinary cases. That change is quiet in the product, but it is what lets concurrent suites run without depending on host-wide serialization for routine configuration behavior.

This sprint attacked the hidden dependency on ambient configuration and current-directory state. The brief named an explicit process context contract, which let the agent work at the boundary where runtime settings are resolved and passed inward. It stayed green because the public command-line behavior could remain unchanged while tests gained injectable inputs. That made later ownership work possible without a broad behavioral rewrite.

Owned Processes

2 agents 20m 45s wall time 119k peak context

Process-owning tests now use support helpers that make global mutation explicit and bounded. The reliability docs still point contributors toward the process guard, environment restore, and unique scratch directory patterns. That makes process ownership auditable in the test suite rather than an invisible condition every run inherits from the host.

This sprint narrowed process locking to cases that actually own global state. It consumed the explicit context contract from the previous sprint and replaced a fixed host-wide lock with a workspace-scoped, bounded ownership model. The boundary stayed green, which fits a test-support change whose main risk was fixture coordination. The result gave timeout and stress work a smaller lock surface to reason about.

Runner Timeouts

2 agents 12m 51s wall time 131k peak context

The committed nextest configuration carries finite slow-test behavior, and `bin/test` wraps fallback tiers with a timeout command. The script reports timeout exits clearly instead of letting a missing preferred runner create an unbounded path. Heavy and default profiles remain separate, so long-running coverage can be bounded without making the ordinary gate inherit its ceiling.

This sprint made hangs a verdict instead of an indefinite wait. Its brief split the work between preferred runner configuration, heavy-profile limits, and fallback script bounds, keeping the agent away from unrelated driver semantics. The facts show a clean pass, indicating that the timeout contract could be added at the runner layer without reopening process-context work. It also supplied the premise for baseline-gate supervision in the next sprint.

Spend Guards

2 agents 14m 39s wall time 171k peak context

Gate execution is now centralized in a runner that owns process groups, records gate subprocesses, emits progress, and can reap work on abort. Baseline and goal verification paths also distinguish clean, red, and no-work outcomes in the current engine. Some timeout documentation from the run has since been revised, but the surviving implementation still treats gate subprocesses as owned work rather than unmanaged shell commands.

This sprint moved from test-runner bounds to Gantry's own baseline-gate supervision. The brief asked for no-progress and total-time concepts to classify a non-terminating baseline gate as infrastructure failure while preserving normal red and green verdicts. It passed green, which shows the orchestration layer could consume the runner timeout contract cleanly. The sprint completed the milestone's chain from individual tests to run-level gate ownership.

Milestone 3

Root Test Tiers

5 tasks 13 agents 1h 8m wall time 194k peak context

The current root workflow still exposes `bin/test`, `bin/test --heavy`, `bin/stress`, nextest profile bounds, and a reliability guide explaining how to use them. The default gate runs ordinary behavioral coverage and feature-boundary checks, while heavy and display-oriented work are explicit. The guardrail test also remains, keeping the tier split from eroding as new tests are added.

This milestone turned the prior isolation work into an operating model for the root gate. It first audited driver coverage, then changed the root runner, added stress evidence, installed a guardrail against accidental real git use, and wrote the contributor guidance. The run facts show green sprint boundaries with extra gate passes around tiering and docs rather than repair. The decomposition worked because each sprint consumed a named contract from the previous one.

Driver Surface Review

2 agents 20m 18s wall time 194k peak context

The current run tests use shared scaffolds and comments to separate real production wiring from pure driver decisions. Later architecture changes make the exact audit edits hard to isolate, but the heavy-surface idea is still present in the tier docs and runner. The suite now has language for why a driver test is real integration rather than simply old.

This sprint finished the audit that earlier milestones had left open before any heavy-tier move could happen. The brief forced real driver tests to justify external repository or process behavior, while displaced decisions had to stay covered through fake-backed paths. It passed green, so the classification work did not need another repair loop. That gave the root runner sprint a defined heavy surface instead of a bulk quarantine.

Root Tier Runner

2 agents 14m 36s wall time 170k peak context

`bin/test` still implements the root tier model. With no flag it runs the ordinary default-feature suite and feature-boundary checks; with the heavy flag it discovers tests marked for the heavy surface. Supporting docs and release scripts refer to that split, so contributors have a command contract rather than an informal convention.

This sprint made the mechanical gate change after the heavy surface had been named. Its brief split default behavioral coverage, stripped feature checks, and explicit heavy coverage into script-level contracts. The facts show the gate ran more than once but stayed successful, which points to careful validation of the runner shape rather than a failed build. The unit was self-contained because it owned the top-level script and its immediate documentation.

Stress Load Standard

2 agents 7m 51s wall time 100k peak context

`bin/stress` still has an evidence mode that requires a module scope and harness-generated load before it will act as closeout evidence. It records metadata and a status table for later analysis, and `scripts/analyze/stress-report.py` preserves those outputs. The reliability guide still distinguishes focused development checks from evidence runs.

This sprint defined what counts as flake closeout evidence under contention. The brief made the harness reject narrow single-test proof for flake-class work and required whole-module scope, full parallelism, repeated sampling, and competing load. It passed green as a focused script and analysis change. The boundary is important because it changes the meaning of a green stress run, not just the command syntax.

Default Git Guardrail

2 agents 6m 2s wall time 90k peak context

The guardrail test still scans the default integration test surface for process-global mutation, sleep-based readiness, weak scratch paths, duplicate stub state, and unauthorized real git subprocesses. Its messages point contributors toward approved helpers or the explicit heavy marker. That keeps accidental end-to-end driver behavior from slipping back into the ordinary gate.

This sprint protected the tier split against regression. Its brief asked for representative default-tier tests that spawn real git outside approved fixtures to fail with a useful message. That was a small and well-contained unit because it only needed to scan test sources and honor known exceptions. It stayed green, making the guardrail a policy check rather than another manual review item.

Tier and Stress Docs

2 agents 8m 27s wall time 87k peak context

The reliability guide still describes the default root gate, heavy tier, display tier, timeout expectations, fixture patterns, and stress closeout standard. A stress result note remains under the analysis docs, tying the guidance to an actual recorded run rather than a bare claim. Later test-policy updates expanded the page, but this sprint's tier vocabulary remains in the current workflow.

This sprint converted the new tier and stress contracts into durable contributor guidance. It came after the runner, stress harness, and guardrail were present, so the agent could document observed behavior rather than planned behavior. The facts show an extra gate pass but no repair, which fits a documentation sprint tied to executable checks. It closed the root-side milestone with a recorded stress result.

Milestone 4

Web Structural Assertions

4 tasks 11 agents 22m 34s wall time 102k peak context

The Rails test tree still has structural assertion helpers, page-rendering tests, pricing and checkout checks against source-of-truth data, download and legal flow assertions, and a quarantine for exact legal copy. Ordinary page prose is no longer the main proof that public pages render. The web gate can protect behavior and structure without treating marketing edits as product regressions.

This milestone applied the same anti-brittleness standard to the Rails app. Its sprints established a structural assertion contract, then converted license access, pricing and checkout flows, and residual copy pins around that contract. The facts show green boundaries with extra gate passes around page structure, commerce assertions, and cleanup, which is where the web suite touched broad integration behavior. The milestone did not reopen root tiering; it consumed the finished root guidance.

Page Structure Contract

2 agents 3m 36s wall time 63k peak context

`StructuralWebAssertions` still defines the page contract used by Rails integration tests. Page rendering checks call helpers that assert app chrome, page identity, current navigation, sections, and link targets. That leaves ordinary body copy editable while keeping the page shape and navigation behavior under test.

This sprint created the web-side contract before converting specialized flows. The brief asked the agent to prove durable page identity, chrome, sections, navigation, calls to action, links, and behavior instead of broad body prose. It passed green with an additional gate pass, which fits a foundational test-helper change touching many page-rendering expectations. The sprint gave later web agents a shared assertion vocabulary.

License Access Structure

2 agents 6m 6s wall time 102k peak context

The current web tests verify license and legal document rendering through app chrome, download links, attachment headers, and nonempty source-backed content instead of broad text snapshots. Related views expose data and links for integration assertions. Later cleanup removed some original license-access lines, but the surviving checks still protect access behavior rather than prose wording.

This sprint converted license and download coverage after the structural contract existed. The brief kept the scope on access states, download actions, release assets, and hermetic behavior rather than explanatory prose. It passed cleanly, suggesting the shared web contract was adequate for this flow. The unit did not need to touch pricing or copy quarantine yet.

Pricing Checkout Forms

2 agents 4m 13s wall time 57k peak context

Pricing and checkout tests still assert live offer metadata, consent handling, hosted checkout redirects, server-owned price data, product metadata, terms versions, and rejection of client-supplied tier data. Download tests verify the demo and personal paths through links and forms. These checks protect the purchase flow without pinning the surrounding marketing text.

This sprint took the commerce-sensitive web surface and kept it separate from general page rendering. The brief asked for prices, product identifiers, consent, validation, submission behavior, and checkout destinations to remain protected through stable sources of truth. The facts show an extra gate pass but no repair, which fits a flow with several app layers but a clear acceptance contract. It left exact wording outside the core proof.

Copy Quarantine Cleanup

2 agents 3m 27s wall time 94k peak context

Exact legal-copy protection now lives in `web/test_quarantine` with regeneration guidance, while default Rails integration tests assert legal pages structurally. The temporary handover document from the run is gone in the current tree. That leaves canonical legal bytes protected when run explicitly and keeps routine web verification focused on behavior, links, and structure.

This sprint closed the web milestone by sweeping residual broad copy pins and moving intentional exact-copy checks out of the default web gate. The brief also required the temporary handover artifact to disappear once its issues were resolved or retired. It passed with an additional gate pass, appropriate for a cleanup unit whose risk was missing a lingering copy assertion. The result completed the Rails side without changing the root-tier work already settled.