Hermetic Test Policy
Hermetic Gate Contract
A post-purge test suite gained hermetic seams, explicit heavy tiers, runtime budgets, and policy text.
2 milestones 10 tasks 23 agents
3 plan 10 execute 12 review 1 fix
214k peak context
124k median execute
3h 20m total 10m 10s per task
1h 14m execute 25m 25s review
63 edits 740 commands
codex harness
This run followed the test-suite purge with a rebuild of the coverage that still earned a place. Its first milestone restored the audit findings without bringing back broad deleted files, and its second milestone turned that restored suite into an operating contract for future work. The result is partly code and partly policy. Rails tests now fail closed around secrets and network access, Rust keeps narrow guards for run-state and operator-facing regressions, and the root gate names the difference between ordinary checks, heavy checks, and display-stack work.
How this walkthrough is structured
Feature
What did the run build and ship?
The live tree still carries explicit Rails service inputs, marker-selected heavy Rust tests, runtime and flake evidence, and agent guidance that tells future work how not to regrow bad tests.
Build
How did Gantry structure the work?
The cut separated hermetic restoration from gate policy, letting fresh agents restore narrow detectors before later agents measured and documented the suite they had produced.
Milestone 1
Hermetic Audit
6 tasks 16 agents 1h 27m wall time 214k peak context
Rails and Rust now both show the restoration work in the tree. The Rails side has explicit service input helpers, test-environment refusal for outbound network use, and restored endpoint coverage. The Rust side has narrow guards around removal refusal, run-state rollback, attachment behavior, absent metrics, and heavy map abort cleanup, with the expensive concurrency detector outside the default gate.
This milestone carried the restoration problem while the later policy work waited. Its brief kept the boundary sharp: restore only the audit findings that were wrongly lost, add seams where surviving tests needed them, and keep machine-dependent coverage out of the default path. The nested sprints all completed, but the interface-observability sprint exposed the hard edge with a red gate and a repair pass. The milestone review then accepted the restored suite before the run moved on.
Rails Hermetic Seams
2 agents 13m 56s wall time 130k peak context
The Rails tree still has a shared helper for setting service inputs and a refusal module that blocks HTTP and raw socket paths in tests. The test environment initializes those inputs to inert values and installs the network refusal after boot. The affected services read `Rails.configuration.x` first and decline ambient credentials or environment fallbacks under test. The hermeticity test proves both the refusal path and the explicit-input path.
This sprint was asked to make the Rails test tier fail closed before any seam-dependent audit coverage came back. That was small enough for a fresh agent because the contract named the service inputs directly: signing keys, public static base, cache purge configuration, and outbound network refusal. Its gates stayed green, so the boundary held at the seam layer rather than leaking into product-flow restoration. Later Rails tests could consume this explicit contract instead of assuming the worker machine was empty.
Rails Audit Restorations
2 agents 9m 29s wall time 124k peak context
The demo API integration test now covers exhausted allowances, replayed or malformed completion credentials, indistinguishable public rejection shapes, malformed requests, rate limiting, and demo-code unlock refusal. Other Rails tests keep structural checks around post rendering, post metadata, build availability, and replay tasks. Those tests state their keys through helpers or pure fixtures, so the coverage is not dependent on encrypted credentials or live network access.
This sprint consumed the Rails seams rather than reopening the deleted files wholesale. Its brief named endpoint unlock rejection, demo allowance exhaustion, content routing, and build-availability detectors as narrow behavior checks. The run kept this work green, which says the seam contract from the prior sprint was adequate for the restored Rails money paths. The unit boundary was product behavior, not test infrastructure.
Rust Refusal Boundary
2 agents 9m 53s wall time 117k peak context
`src/commands.rs` still contains an inspectable live-owner removal decision and tests for the explicit live override. The command path refuses live-owned removal unless the caller supplies the override, and process-safety cleanup reports unverifiable states instead of silently tearing down a run. The behavior is now covered where the decision is made, which keeps the CLI fail-closed without depending on process-table luck.
This sprint isolated remove-refusal decisions from command termination. The brief made that a pure decision boundary: live owners and unverifiable process safety had to be testable without making the test process exit. Its gates stayed green, so the extraction was a contained refactor rather than a broad command rewrite. That gave later safety restorations a cleaner command surface to lean on.
Rust State Safety
2 agents 14m 35s wall time 214k peak context
The live tree has default-tier tests and code comments around red gate classification, review verdict absence, resume-time cleanup, path-scoped resets, stash ownership, and shared-checkout safety. Worktree rollback helpers distinguish the run worktree from the shared main checkout and refuse ambiguous stash situations. MCP and goal helpers also keep structured missing states visible. Together those pieces guard the failure paths where a careless change could damage user work or misreport a run.
This was the broadest Rust restoration in the first milestone. The brief grouped related red-path risks: orchestration state, rollback behavior, red verdicts, resume identity, stash ownership, and checkout-write refusal. Because each item was a narrow detector for a concrete audit finding, a fresh agent could add coverage across several modules without owning a redesign. The gates stayed green, so the slice held as a coverage restoration rather than a new feature build.
Rust Interface Observability
3 agents 13m 46s wall time 160k peak context 1 fix
Harness code now parses usage-limit reset hints through injectable parsers, so tests can exercise wait guidance without real time. Attach-client and MCP paths report no-live-owner and disconnect cases as structured failures rather than mutating the owner. Stats rendering and CLI code preserve missing measurements as absent data instead of formatting them as a measured empty value. Some of the exact additions have since been surrounded by later harness work, but the absent-value and attach behaviors remain visible in the current files.
This sprint covered user-facing observability rather than core state mutation. The brief asked for reset-wait guidance through injected time, ownership-sensitive attach behavior, and absent-value reporting. The run found its stress point here: the gate went red after review and a fix pass was required before the sprint settled. That places the difficulty at the interface between meaningful absence and rendered output, not at the basic harness path.
Heavy Map Concurrency
2 agents 10m 27s wall time 139k peak context
`driver_map.rs` still carries a heavy-marked abort test that starts a real map worker pool, raises an abort with workers in flight, and checks that worker process groups are reaped. The root test script later discovers that marker rather than naming the test in a central list. The default suite can skip the case while the explicit heavy tier can still run the real concurrency path.
This sprint kept the only truly concurrent restoration out of the ordinary gate. Its brief required a real map worker pool and an operator abort, which made the work heavy by nature. The sprint stayed green because it was scoped as a single heavy detector, not as a general display or stress tier expansion. That boundary mattered later when the gate policy replaced central membership with test-owned markers.
Milestone 2
Gate Baselines
4 tasks 11 agents 1h wall time 150k peak context
The operating contract from this milestone is still visible in the live tree. `bin/test` discovers heavy tests from `heavy:` ignore markers, the reliability guide records the gate shapes, runtime budgets, stress workflow, flake baseline, and rejected antipatterns, and the agent indexes link that guide directly. The merge trouble did not erase the feature evidence: the authored policy and gate code are present at HEAD.
This milestone assumed the restored suite was present and shifted the problem from coverage to operation. Its nested sprints moved heavy selection onto test-owned markers, measured the resulting gates, refreshed flake evidence, and published the policy where agents read it. The sprint gates stayed green, but the final merge path hit repeated integration trouble after the work was complete and never reached a clean automatic merge in the recorded timeline. The decomposition still separated implementation, measurement, evidence, and documentation cleanly.
Heavy Marker Gate
2 agents 20m 40s wall time 150k peak context
`bin/test` now scans Rust source for ignore reasons beginning with `heavy:` and resolves each marker to an actual listed test. The default gate runs ordinary behavioral checks and excludes ignored heavy cases, while `bin/test --heavy` selects the marked cases exactly. The flaky-antipattern guardrail also understands declared heavy surfaces, so real git or subprocess use has to be either local fixture code or an explicit heavy exception.
This sprint replaced a fragile central heavy-test registry with a marker contract owned by the tests themselves. The brief made the dependency explicit: later runtime and flake measurements would only mean anything once the tiers selected the intended work. The sprint completed green, so the marker model was established before any baseline prose depended on it. That made this unit a gate-structure change rather than an audit-restoration task.
Runtime Baselines
2 agents 9m 44s wall time 118k peak context
The reliability guide now names the root default gate, the explicit heavy gate, the Rails gate, and the display-stack boundary. It also records the measured command shapes, slow-tail analysis, and budget rules that future agents are supposed to consult before adding slow coverage. `bin/test` itself carries bounded fallback behavior and the marker discovery path, so the documentation matches an executable gate rather than a stale description.
This sprint was a measurement and budget pass over the marker-based gate. Its brief warned against widening into hosting, coverage ratios, quarantine, or display-stack mechanics, so the work stayed on the root gate, heavy tier, Rails gate, and cheap local-only trims. The gate stayed green, which meant the recorded budgets were grounded in a working suite shape. This unit produced operating evidence for later policy prose rather than more regression detectors.
Flake Guardrails
2 agents 11m 49s wall time 74k peak context
The flake snapshot files remain in `docs/analysis/flaky-test-snapshots`, and the reliability guide links the post-purge baseline. The guide distinguishes focused development samples from closeout evidence and requires whole-module or otherwise approved scopes for closing a flake class. The existing guardrail still scans for process-global mutation, sleep-based readiness, brittle scratch paths, shared mutable fixtures, and undeclared real git subprocess use in default-tier tests.
This sprint refreshed reliability evidence under the settled tier contract. The brief allowed fixes only for real flake classes surfaced by the baseline and allowed guardrail expansion only for observed pattern classes. The run stayed green, and the fresh evidence did not require a behavior fix or new guardrail class. That kept the unit honest: it recorded bounded evidence instead of claiming permanent flake absence.
Policy Docs
2 agents 5m 28s wall time 76k peak context
The root agent index links the reliability guide alongside the main verification commands, and the web agent index repeats the Rails-specific no-network and no-credentials rule. The shipped prompts still tell agents that tests must earn their place, avoid machine dependence, and prefer deleting bad tests over weakening them. `docs/analysis/test-reliability.md` now carries the concrete rejected-antipattern list and stress workflow, so the policy is available where future work begins.
The closeout sprint turned the gate and evidence work into guidance for the next agent. Its brief said not to re-litigate the policy, only to make the current rules discoverable, remove stale names, and express the purge lessons as properties. The sprint completed green, so the documentation pass did not disturb the gate it was describing. This was the handoff layer for the whole run.