Riding Out a Transient API Outage
Gantry tells a momentary upstream blip from a real failure and waits it out instead of halting the run.
A Gantry build can run for hours, and every agent it spawns reaches the model through a single runner. When the upstream API answered one of those calls with a momentary "overloaded" response — the kind that clears on its own within minutes — Gantry could not tell it apart from a genuine error. It recorded a failure, stopped the pipeline, and ended the whole build, undone by a server hiccup that would have passed if anyone had simply waited.
This job taught the runner the difference. A temporary upstream failure is now recognised as its own class and waited out on a bounded, escalating schedule: try again at once, then after ten minutes, then after an hour, and only then give up. Because every agent call passes through that one runner, the whole pipeline became tolerant of these blips at a stroke. Both the terminal interface and the headless log show a backing-off run explicitly, so a build that is waiting no longer looks like one that has hung.
Build
The work split into three sequential phases through the single point every agent call passes: recognise a transient failure, then act on it, then make the waiting visible — each phase leaving the test suite green on its own.
The load-bearing decision was to separate recognising a transient failure from doing anything about it. The first phase added the new outcome class, threaded captured stderr into each harness's parse step, and built the allow-list matcher — then immediately collapsed the transient result back into an ordinary error, so run behaviour stayed byte-for-byte identical to before. That let the entire classification surface land and be unit-tested while no retry loop yet existed. The second phase was then a contained swap: replace the collapse with a real loop against a classifier already proven correct. A cut that built detection and retry together would have had to test both at once, with no green tree in between; this ordering gave each piece a proven floor before the next leaned on it.
The one thing source code could not assert was the exact text each live CLI prints on an overload — so the brief required seeding the matcher from a real captured transcript rather than a guess, and kept the allow-list conservative by rule: a missed signature reproduces the old halting behaviour, never a regression and never an endless retry on a genuine bug. That took the boundary's sharpest risk off the table before it could bite.
The run bears this out. All three sprints ran green — no gate sent one back, no review forced the remaining work to be re-planned. On a change that reaches into four harnesses and the core runner, a clean pass is the evidence that the phasing dissolved the difficulty in advance rather than the sign of a task that was never hard.
Feature
Gantry now carries a distinct agent-outcome class for retryable upstream failures and a bounded backoff loop inside the runner; both still stand at today's HEAD, and the loop has since grown to cover usage-limit waits on the same machinery.
Before this job, a temporary upstream blip surfaced as a plain error, flowed straight to the stop path, and ended the run — there was no way to say "try again shortly" as distinct from "this request is wrong." The answer is a new Transient outcome produced only at the harness parse step, a shared case-insensitive matcher over an allow-list of overload and gateway signatures, and a retry loop that wraps each attempt under the full per-spawn timeout while the backoff wait sits outside it, abort-aware so an interrupt during a long wait returns promptly. A schedule configurable through LOOM_RETRY_DELAYS drives it, and a dedicated event renders the backoff in both front-ends.
Standing in the tree today, the design holds. The matcher module is still present, and has become the shared home for the other output-classification helpers as well — the ceiling-detection and reason-line trimming that every harness parse relies on. The Transient variant and the retry loop live in the runner, most of the lines the job introduced survive at HEAD, and the transient class is still kept invisible above that runner: callers see either success or a collapsed error, never the retry itself.
The clearest drift is growth, not decay. The same loop now also handles usage-limit reset-waits — a separate category with its own bounded counter, drawn on when a failure carries the exact seconds until a usage window reopens — reusing the retry machinery rather than standing a parallel one up beside it. One file the job edited no longer exists under its old name; its callers folded into the surrounding modules as the engine moved on.