Agent failure recovery: when a run gets stuck

It's 2 a.m. You kicked off a release before bed — twelve stories, a few of them parallel — and went to sleep trusting the machine. You wake up to a queue that hasn't moved in six hours. One story tried to merge, hit a conflict, and has been quietly retrying ever since. Another agent stopped emitting output around 3 a.m. and has been "running" against a dead process the whole time. A third looped on the same three edits until it ran the API bill up. Everything downstream of those three is stuck behind them, and nothing tells you which one wedged the run.

That's the failure mode nobody puts in the demo. Autonomous coding works right up until something goes wrong — and what happens after it goes wrong is the entire difference between a tool you can leave running and a tool you have to babysit. Agent failure recovery is the unglamorous half of autonomy, and it's where most tools quietly fall apart.

This post is about how Trinity treats failure as a first-class state instead of an accident: it names the failure, measures what it blocks, drafts a way forward before you even open the story, parks hung runs at a recovery gate instead of pretending — and now detects stuck or looping runs on its own and stops them with a reason, without losing the work that already committed.

Most agents fail in one of three ugly ways

Watch enough autonomous runs and the failures rhyme. There are three shapes, and they're all bad.

The silent stall. The agent hangs — a subprocess dies, a network call never returns, the harness loses the thread — and the run just sits there. No error, no progress, no signal. From the outside it looks identical to "still working." You find out hours later, when you happen to look.

The runaway loop. The agent keeps acting but stops converging. It edits the same file, re-runs the same failing test, re-plans the same step, certain it's about to finish. This is the failure mode the field talks about most — practitioners call it loop drift, where an agent "keeps thinking, calling, retrying, or re-planning long after it has stopped making progress." Left alone, it burns tokens and wall-clock until something external kills it.

The burned retry budget. A failure that retrying can never fix — a merge conflict, a closed PR, an expired credential — gets treated like a transient blip. The harness retries it, the same wall reappears, it retries again, and the whole budget evaporates against a problem that was never going to resolve itself.

The common thread isn't that things break. Things always break. It's that the default response to breakage is to stall, spin, or silently exhaust the budget — and then leave you to reconstruct what happened from a log.

That last column isn't one feature. It's four releases of treating failure as something the system owns, not something it hands back to you. Here's how it got built.

Failure is a state, not an accident

The first move was the simplest and the most important: give failure a name.

Before, a story that couldn't get through would just stall — indefinitely, indistinguishable from work in progress. The groundwork release changed that. A story that repeatedly fails, hits a merge conflict, or has its PR closed out from under it now lands in a clear Failed state instead of hanging. And a Failed state is something you can do things to: open Diagnose to investigate the run in a read-only chat, then Retry to clear the failure and send the story back through the pipeline.

That sounds modest. It's the hinge the whole system swings on. Once failure is an explicit state in the story lifecycle rather than the absence of progress, every other capability becomes expressible — you can measure it, draft against it, detect it, and recover from it. You can't build recovery on top of "it's probably still working."

The diagram above is the shape the rest of this post fills in. Running, with work committing as it goes. A detector that catches the real failure modes by name. A recovery gate that parks the story instead of abandoning it. A drafted resolution — retry, reshape, or remove — one click away. And a path back into the pipeline that resumes from what already landed.

Measure the blast radius before you panic

A failed story is rarely just one story's problem. It's a node in a dependency graph, and everything downstream of it is now stuck too. The question that actually matters at 2 a.m. isn't "did something fail" — it's "how much did this take down, and is anything still moving?"

So Trinity measures the blast radius the moment a story fails. The story's row on the Run page explains what went wrong in plain words and carries a "blocking N stories" badge when other work is waiting behind it. The urgency of the surface matches the size of the damage: a failure that blocks nothing gets a quiet note; a failure that blocks four stories gets a badge; a failure that leaves nothing else able to progress raises a prominent banner that names the exact story wedging the run — and sends a notification, so you find out even when you're asleep.

That's the difference between "a run failed" and "this story failed, it's blocking these four, and here's why." You can see the same dependency relationships in the dependency graph — the blast radius is just that graph, read backward from the break.

A drafted resolution, waiting before you open the story

Naming the failure and sizing it is half the job. The other half is: now what? This is where most tools stop and hand you a stack trace.

Trinity drafts the fix. The moment a story fails, the architect reviews what happened and proposes a concrete resolution — with reasoning and the exact changes — so by the time you look, a reviewable option is usually already sitting there. There are three shapes it can take:

1. Diagnose what actually broke

   Trinity classifies the failure into a plain-language reason — "merge conflict", "PR closed externally", "authentication failed" — instead of an internal code. Open the read-only diagnostic chat to dig into the specific run without touching anything.

2. Read the drafted resolution

   The architect proposes one of three moves: retry it as-is, reshape the story so it can land, or remove it and rewire the stories that depended on it. The proposal shows up on the failed story itself and right on the Run page, with the reasoning and the exact changes spelled out.

3. Approve, or steer

   One click approves the drafted change or dismisses it. Or skip the binary entirely: give feedback in plain language in the diagnostic chat and re-run with your guidance, without editing code yourself.

4. Resume from where it is

   Approve and the story re-enters the pipeline — picking up from the work that already committed, not starting over.

The "reshape" and "remove" options are the ones that matter most, because they're what a human would actually do. Sometimes the story was wrong, not the run — it needs rewriting before any retry will succeed. Sometimes the right answer is to remove it and rewire the four stories waiting on it, atomically, without leaving the dependency graph half-broken. The architect can do both, and it shows you the ripple before you confirm.

Marking a stuck story failed never deletes the work

There's a specific fear that keeps people from letting an agent fail honestly: if I mark this dead, do I lose everything it did? Because half-finished work that vanishes is worse than a stalled run.

So the rule is absolute, and it's worth stating plainly.

What makes that safe isn't a backup. It's that the work was never sitting unsaved in the first place. The implementer commits incrementally as it works through a story, so progress lands on the branch continuously rather than in one final commit at the end.

That's the quiet engineering decision underneath everything else. Recovery only feels safe because interruption is cheap, and interruption is cheap because the work commits continuously. You can read more about how workers and retries behave in the execution docs.

The recovery gate: park, don't pretend

Diagnose-and-retry is great when you're the one who noticed. But the worst failures are the ones where the agent can't even raise its hand — it hangs without reporting back, times out, or quietly runs out of retries. The old behavior was to mark those failed (eventually) and move on. The better behavior is to park them.

When a story hits a hard failure like that, Trinity now routes it to a recovery gate instead of silently failing. It's one of Trinity's typed execution gates — the same machinery that pauses the pipeline for human judgment, applied to failure. The story parks with its committed work safe, a diagnostic chat ready, and the architect's suggested fix usually already waiting. You investigate and re-run with guidance, or mark it failed and move on.

The other half of that change is about when to give up. A failure that retrying can never fix is now caught after a single attempt instead of burning through every retry first. There's no point throwing three more runs at a closed PR or an expired credential — the wall isn't going to move. Trinity tells the difference between "try again, this was transient" and "stop, this is terminal," and only the transient ones get the retries.

A run that can't make progress shouldn't get to pretend it's making progress. Park it, name it, and draft a way out.

Now Trinity catches the stuck runs itself

Everything so far still leans on a failure that announces itself — an exception, a timeout, a raised hand. The hardest cases don't announce anything. The agent is still "running." It's emitting tokens. It just isn't getting anywhere. That's the silent stall and the runaway loop, and the latest step closes the gap on both.

Trinity now watches runs for three patterns and stops them on its own:

• Stalls — a run that goes genuinely silent, with no real activity, gets detected as hung and stopped, rather than holding a worker forever against a dead process.
• Loops — a run that keeps repeating the same actions is flagged as "run stuck in a loop" and halted before it churns through the budget.
• No convergence — a run that keeps working but never closes in on done is stopped with the reason "run not converging."

In every case the run is stopped and parked for diagnosis — dropping it straight into the recovery flow above, with a clear reason attached — instead of hanging indefinitely or silently draining retries. The detection is what the field keeps prescribing as the fix for loop drift: external, deterministic loop guardrails that watch behavior and enforce termination, rather than trusting the model to notice it's stuck. The model is the worst-positioned thing to judge whether the model is looping.

The subtle part is not over-triggering. A long test suite or a slow build isn't a hung run — it's a busy one. So a run is judged by whether it's still doing real work, not by a fixed time limit. A lengthy-but-active job keeps going; only a genuinely silent or circular one gets stopped. The goal is to catch the runs that are wasting your time without cutting off the ones that just need it.

Where most autonomous agents stop

To be fair about the landscape: detecting that an agent is stuck is a known, hard problem, and the ecosystem is actively working on it. The standard advice across 2026's agent-engineering writing is to bolt on external loop guardrails and execution limits — caps on iterations, deterministic stop conditions, state validation. That advice exists because the default behavior, across most autonomous coding tools, is still to stall silently, loop until something external intervenes, or retry an unfixable error until the budget is gone.

Trinity's difference isn't that it invented loop detection. It's that detection is wired into a system that already knows what to do next. Catching a stuck run is only useful if the run lands somewhere — a named Failed state, with its blast radius measured, its work safely committed, and a drafted resolution waiting. Detection without recovery just gives you a faster way to find out you're stuck. Recovery without detection means you find out too late. Trinity does both, on the same substrate, so a 2 a.m. failure becomes a 9 a.m. one-click decision instead of a morning of forensics.

Back to 2 a.m.

Run the same night again. The story that hit a merge conflict didn't retry into the void — it stopped after one attempt, landed in a Failed state with "merge conflict" written plainly, and the architect drafted a reshape. The agent that died around 3 a.m. didn't hold a worker until morning — Trinity detected the stall, stopped it, and parked it. The one that looped got caught as "run stuck in a loop" before it ran the bill up. All three are sitting at a recovery gate with their committed work intact and a proposed fix attached. The banner already told you which one wedged the run.

You don't reconstruct anything. You read three drafted resolutions, approve two, reshape one, and the queue starts moving again — picking up from exactly where each run left off.

That's the whole idea: failure isn't the end of an autonomous run, it's a state the system is built to handle. See the full set of changes in the 0.3.7 changelog, and the automatic stuck-and-loop detection in 0.3.15.