Troubleshoot Before Halt
Recovery Gets the First Pass
A recovery rebuild that moved repair, halt routing, pause, and plan cleanup into the shipped flow.
4 milestones 13 tasks 26 agents
5 plan 9 execute 18 review
164k peak context
132k median execute
2h 52m total 10m 53s per task
58m 18s execute 1h 6m review
116 edits 1.1k commands
codex harness
This run took a cluster of recovery plans that had grown around stopped builds and folded them into the main orchestration path. The digest divides the work into repair unification, halt-hook policy, operator pause, and cleanup, with each milestone planned as small sprints that changed pattern language, engine routing, prompt contracts, display controls, and the plan queue. The build record is mostly green at the sprint level, so the interesting cut is not a dramatic repair loop. It is the way the run kept policy boundaries separate: repair stayed a troubleshoot responsibility, halt hooks became declared pattern policy, pause remained a deliberate operator stop, and cleanup waited until the new behavior had names in the tree.
How this walkthrough is structured
Feature
What did the run build and ship?
The tree now carries a shared troubleshoot prompt, pattern-declared halt hooks, pause and resume controls, compatibility wording for retired stages, and a live plan queue that treats the recovery cluster as landed.
Build
How did Gantry structure the work?
The work was split by recovery boundary, letting mostly green sprint rows carry prompt repair, halt policy, pause semantics, and cleanup without mixing operator intent with failure handling.
Milestone 1
Unified Repair Ladder
4 tasks 10 agents 53m 50s wall time 169k peak context
Current sprint and milestone patterns route red work to troubleshoot, and the prompt now describes repair, retry notes, and handover as one contract. The retired fix and investigate stages remain in the domain enum as compatibility tokens, not active shipped repair routes. Documentation and prompt inventories still explain that distinction for archived journals and frozen material.
This milestone carried the core recovery contraction: remove the separate repair path and make every repair rung speak through troubleshoot. Its brief gave the agent a tight invariant to preserve, because historical records still had to parse while new patterns stopped emitting the retired vocabulary. The measured run did not record a recovery or re-plan here, so the boundary held as a straight implementation pass across the nested sprints.
Unify Repair Ladder
1 agents 20m 54s wall time 115k peak context
The shipped sprint pattern now sends both normal and post-review repair positions through the troubleshoot prompt and stage. The position clauses carry the escalation rules, including the post-review revert option and the rule that a handover before the ladder is spent does not have to stop the unit. Troubleshoot disposition, note, and handover channels are declared state instead of ad hoc files.
This sprint had the riskiest repair boundary: ordinary red gates, post-review red gates, stale channel state, and first-rung escalation all had to move at once. The brief kept that work inside the inner-unit ladder, which made it small enough for a fresh agent to reason about without also touching halt policy. Its checks ran cleanly in the measured record, so the later sprints could treat the contract as established.
Port Loop Repair
2 agents 7m 9s wall time 132k peak context
The Ralph loop pattern now uses troubleshoot disposition and note channels instead of a separate repair vocabulary. Its repair block names troubleshoot as the stage and prompt, so the non-inner loop no longer depends on retired fix emission. The standing-loop behavior remains recognizable while sharing the same signal shape as sprint repair.
After the inner ladder moved, this sprint had a narrower job: bring the standing loop repair path under the same contract. That cut isolated generic prompt rendering from the sprint-specific path, so a fresh worker only had to prove the shared prompt could be filled outside the usual unit pipeline. The fact record shows no recovery for the sprint, which fits a port bounded by an already written contract.
Keep Reporting Compatible
2 agents 8m 20s wall time 100k peak context
Compatibility is still visible in the source: retired fix and investigate stages parse and render for journals and replay records, while active repair work is counted through troubleshoot. The artifact reader and statistics surfaces retain repair activity as a semantic concept instead of tying it only to the old stage name. Current docs describe old stage values as archival support rather than new protocol.
This sprint protected old evidence while new runs changed their vocabulary. The brief separated compatibility parsing from live emission, which kept the agent from treating a historical stage rename as a product rewrite. It passed without a recorded recovery, leaving the documentation sprint a settled reporting contract to describe.
Refresh Docs Artifacts
2 agents 9m 4s wall time 169k peak context
The prompt README, agent-prompt reference, pattern manual, and pattern snapshots now describe one troubleshoot repair ladder. They also preserve the operator-facing warning that stale global prompt mirrors may still contain retired files until reset. The generated pattern artifacts in the tree match the current pattern sources rather than a hand-edited account.
This was the consistency pass for the repair milestone. The brief deliberately pushed prose, prompt catalog entries, pattern comments, and generated snapshots after the behavior had landed, keeping the agent out of speculative documentation. The run facts show a green boundary, so no later re-plan had to revisit the repair story.
Milestone 2
Halt Policy Routing
4 tasks 10 agents 1h wall time 158k peak context
The build pattern now declares an on-halt hook, and the interpreter has a halt-hook context that carries failure kind, identity, and reason into policy recovery. Resolved hook outcomes only resume after engine verification, while handover outcomes stop resumably with a report. Pattern docs now distinguish this declared policy from first-line support and deliberate stops.
This milestone expanded recovery from red gates to failure-class halts, which made policy routing the main boundary. The outer row was entered more than once in the measured record, but the child sprints themselves show no recovery entries, so the stress stayed around integrating the milestone rather than repairing a single sprint. Its decomposition kept declaration, dispatch, default policy, and documentation in separate passes.
Declare Halt Hook Contract
2 agents 12m 45s wall time 148k peak context
Pattern headers can name an on-halt block, and the pattern documentation describes that declaration as policy data owned by the runnable pattern. The interpreter builds halt-hook context for step and check failures, including the original reason and failing identity. Tests in the current tree cover validation and the distinction between declared termination and engine-forced halts.
This sprint was a language-contract cut, not a recovery behavior cut. It asked the agent to add the header declaration and template-ready failure context while leaving patterns without the declaration unchanged. That made later dispatch work possible without giving the first sprint authority to decide stop policy.
Dispatch Halt Outcomes
2 agents 16m 44s wall time 158k peak context
The interpreter dispatches a declared halt hook once, records context for the hook block, and degrades to the original halt when the hook cannot prove a valid outcome. A resolved outcome re-enters through the ordinary resume-shaped path only after a green check inside the hook. A handover outcome produces a resumable stop with the declared report artifact.
This was the runtime dispatch sprint, where the hook stopped being metadata and became a guarded attempt before surfacing a halt. The brief narrowed the agent's authority to declared outcomes, green verification, handover reports, and recursion refusal. The measured sprint did not need a recorded recovery, which matters because the behavior itself is about only recovering when the engine can prove it.
Route Policy Through Troubleshoot
1 agents 17m 57s wall time 135k peak context
The default build pattern opts into halt-hook recovery before first-line support surfaces covered failures. The sprint pattern sends executor-declared blocked work into troubleshoot, treats missing or unreadable review verdicts as failure, and keeps reviewer-declared blocked as a direct operator stop. Pattern comments and reference docs describe those routes as policy rather than hardcoded engine exceptions.
This sprint applied the hook contract to the shipped build policy and to reviewer and executor routing. Its boundary was policy rather than syntax: decide which failures deserve troubleshoot, which blocked verdicts remain deliberate stops, and when verification is needless. The run record is green for the sprint, so the policy table landed without a later repair loop.
Document Recovery Boundaries
2 agents 4m 56s wall time 110k peak context
The troubleshoot prompt now includes halt context placeholders alongside ordinary gate-repair material. The prompt reference explains machinery agents versus policy agents, verified hook outcomes, handover reports, and deliberate-stop exceptions. The generated reference data includes the same prompt and routing description for site and documentation consumers.
This sprint closed the halt-routing milestone by updating the prose contract for the agents that now receive hook context. Because the implementation was already split across declaration, dispatch, and default policy, the documentation worker could focus on authority boundaries and placeholder completeness. The measured record shows no recovery, so the wording pass did not uncover a structural gap.
Milestone 3
Pause Controls
3 tasks 7 agents 56m 31s wall time 126k peak context
Gantry now has pause-now and pause-at-boundary commands, a paused stop classification, a pause-pending event, and resume-in-place controls in the TUI. The engine treats pause as deliberate operator state, so it bypasses halt hooks, troubleshooting, and generated stop summaries. The live monitor exposes a grouped pause and resume cluster with owner-aware enablement.
This milestone carved pause out as operator intent instead of another halt to troubleshoot. Its child sprint ledger is thinner than the surrounding milestones in the fact tier, but the top-level milestone finished green and its briefs show a clean progression from engine contract to boundary wait to interface controls. That ordering kept process termination, run classification, and user controls from landing as one tangled change.
Stop Contract
1 agents 21m 20s wall time 98k peak context
The engine carries an OperatorPause latch, pause-now commands, and stopped reasons marked as operator pause. Agent execution checks that latch and reaps the subprocess group while leaving uncommitted work available for inspection. Finish and stop-card code recognize operator pause as a deliberate resumable stop rather than a failure that should trigger recovery or generated diagnosis.
This sprint established the pause-now contract at the lowest level: kill the in-flight process tree, preserve the worktree, and classify the stop as resumable operator pause. The brief kept it away from UI work so the fresh agent could concentrate on process and run-state semantics. The run facts show the sprint's stages completed even though the child finish row is not isolated with a commit, so the feature evidence has to come from the tree.
Boundary Pause
1 agents 17m 42s wall time 76k peak context
The pattern interpreter checks for boundary pause between steps and returns an operator-pause outcome before more work is dispatched. The engine emits a pause-pending event while the current stage continues, and pause-now can still escalate the pending boundary request. Resume follows the existing fresh-assessment path instead of a special unpause routine.
This sprint completed the engine-side pause contract by adding the wait-at-boundary mode. The brief narrowed the problem to recording intent, letting current work finish, and stopping before the next step begins, which is the right grain for a memoryless worker. Like the preceding pause sprint, its individual finish is not cleanly separable in the fact record, but the current implementation shows the contract landed.
Pause Controls
2 agents 11m 48s wall time 126k peak context
The TUI registers pause-at-boundary, pause-now, and resume-in-place actions, exposes them in the Run menu, and draws a compact status-line control cluster. Enablement follows ownership, liveness, pending pause, and resumability, and clickable glyphs dispatch the same engine commands as keyboard and menu paths. A running pending-boundary pause renders as a pausing state while still allowing pause-now.
This sprint put the completed pause contract into the live control surface. The brief asked for one action model across menus, keys, clicks, and enablement, so the agent could wire controls without redefining pause semantics. It passed without a recorded recovery, which left the UI as a thin layer over engine commands rather than a separate pause path.
Milestone 4
Plan Retirement Cleanup
2 tasks 7 agents 30m wall time 164k peak context
The active documentation now speaks in the current recovery vocabulary, and compatibility-only references are labeled as such. The live plan queue no longer presents the absorbed recovery plans as pending work. The archive and handover material point readers back to the consolidated recovery record instead of stale intermediate plans.
The final milestone was intentionally sequenced after behavior and controls existed. Its brief asked for a consistency sweep, not another protocol change, so the agents could compare active text against the as-built recovery model and then retire absorbed plans. The measured record is green, and the milestone boundary stayed narrow enough that cleanup did not trigger re-planning.
Recovery Language Sweep
2 agents 10m 53s wall time 164k peak context
Prompt docs, pattern docs, generated reference data, and diagrams describe troubleshoot as the current repair and halt-recovery vocabulary. Retired stage names and prompt files are framed as compatibility for old journals, replay records, or stale mirrors. The live reference also names halt-hook context and deliberate pause exceptions in the same recovery model.
This sprint swept the active prose surface after the implementation milestones had settled. Its scope was broad in files but narrow in judgment: remove active claims about retired prompts, channels, stage emission, and reviewer pass-through behavior while preserving historical compatibility notes. The green fact record suggests the sweep matched the already shipped contracts rather than exposing new behavior work.
Plan Queue Retirements
2 agents 11m 41s wall time 105k peak context
The live plan index lists the recovery work under recently archived material and says the absorbed recovery plans are no longer pending. Superseded live plan files named by the consolidated recovery plan are gone from the active queue, while archived records keep the landed design trace. A reader now finds the recovery cluster as completed history rather than future work.
This sprint handled the plan queue after the vocabulary sweep, which kept repository housekeeping from preceding the proof that the replacement design existed. The brief gave the agent concrete artifacts to remove or archive and a simple correctness test: the live queue should describe pending work only. It completed without a recorded recovery.