Ran Fine · Automation Reliability Audit · Sample

Five silent failures in a production automation stack

Every one of these ran green. No error workflow fired, no alert was raised, and nothing in any log said the word "failed". They were found by looking from outside the system rather than asking it how it was doing.

Subject: self-hosted multi-agent automation stack Scope: 6 scheduled jobs, 3 external APIs Window: 25–27 August 2026 Method: execution records, job state, provider counters

Why this report exists

This is my own system

Audits are usually confidential, so a prospective client can't see one before buying. This is the same report, run against my own production stack, published in full.

It has the shape you'd receive: what's fragile, exactly how it fails, what it costs when it does, and what breaks first. The findings are real and every one carries its evidence.

The headline finding: a daily research job silently produced nothing for three consecutive days. Every run reported success. Nothing alerted. It was discovered by reading the job records directly, not by anything the system said about itself.

Findings

Ranked by what breaks first

01

A daily job delivered nothing for three days while reporting success

Critical
What happensThe job completes, every stage returns done, and the delivery field is written empty. The output exists (it sits in the stage artifact), but the step that selects which stage is the report picked the wrong one and delivered nothing.
Evidence
status: done
final_briefing length: 0
  research_discourse   done   art=9012
  qualify              done   art=5820
  briefing             done   art=4519   <- the report, undelivered
  store_memory         done   art=746
Why nothing caught itEvery check in the system asks "did it finish?" and the answer was yes. No check asked "did anything come out?"
CostThree days of a daily research job, plus the API spend of producing reports nobody received.
FixAlert on an empty deliverable as its own condition, separate from failure. Fixed and verified: the selector now returns 4,519 characters.
02

A hung job stays "running" forever and never alerts

Critical
What happensA job stalls on its first stage. It never completes, so the completion check never fires. It never errors, so the failure check never fires. The record reads running indefinitely, and when the host process restarts, the worker dies leaving an orphan that nothing will ever resolve.
EvidenceObserved twice, three days apart. On 27 Aug a job sat at running for 27 minutes on stage one with all five artifacts empty; a comparable healthy run completes in 566s. The earlier instance was annotated by hand: "never finalized because the host process exited."
Why nothing caught itMonitoring polled for done or failed. A third state (neither) was invisible to it.
CostA full day of missed output per occurrence, with no signal that anything was wrong.
FixWall-clock stall detection: a non-terminal job older than a threshold raises a high-priority alert once. Implemented, with tests covering both the stalled case and the still-healthy-but-slow case.
03

A credential in the wrong variable produced 403s that read like an expired key

High
What happensTwo search providers have near-identical names and incompatible auth. A key belonging to one was stored under the other's variable and sent with the wrong header. The provider returned 403 Unauthorized, indistinguishable from a revoked key.
EvidenceThe stored value was the correct 64-character format and looked entirely valid. It failed for weeks and was repeatedly diagnosed as a billing or account problem. The provider's own error text was the only thing that distinguished them.
Why nothing caught itA present, well-formed credential passes every "is it configured?" check. Only using it reveals the mismatch, and the failure was swallowed by a fallback.
CostWeeks of degraded search across three scheduled jobs, and several hours of misdirected debugging.
FixA credential guard that refuses to send a key whose shape doesn't match its variable, before it reaches the wire.
04

Quota exhaustion degrades search silently, and the degraded mode ignores the filters

High
What happensThe search provider allows 250 queries a month. When it runs out, the chain falls back to a headline feed that ignores site: operators entirely. Jobs that depend on searching specific domains keep running and keep returning results, just not from the domains they asked for.
EvidenceMeasured per job by reading the provider's counter before and after a single run: 22 searches for one job, ~36 for another. Against a 250/month allowance and two daily jobs, exhaustion is arithmetic, not bad luck.
Why nothing caught itThe degraded path returns results. Something that returns plausible output is far harder to notice than something that returns nothing.
CostTargeted research becomes untargeted, with no visible change in behaviour.
FixA second real provider with a 2,500/month allowance inserted ahead of the headline fallback, plus a balance check that alerts when projected burn exceeds the remaining month.
05

A truncated hand-off caused the next stage to invent a fact

Medium
What happensOne stage's output hit a token ceiling and was passed downstream ending mid-word. The receiving stage could tell the input was broken but not why, went looking for the missing half using other tools, and produced a report containing a person's name that does not exist.
EvidenceThe hand-off arrived as "Two thre…". The resulting report attributed a quote to a named author; fetching the source thread showed no such user among 20+ participants.
Why nothing caught itTruncation was treated as a completed reply. Any stop reason that wasn't a tool call counted as "finished", including running out of budget.
CostA fabricated fact in an output intended to be acted upon: the most expensive kind of silent failure, because it is confidently wrong rather than absent.
FixDetect the budget-exhausted stop reason, raise the ceiling, and stamp the output with an explicit truncation notice so downstream stages report the gap instead of filling it.

Cost per run

What each job actually spends

Measured by reading the provider's counter immediately before and after a single execution, not estimated.

JobFrequencySearches / runMonthly drawStatus
Research scandaily~36~1,080over budget
Lead scandaily22~660now ~0
Clone scanwas dailyunmeasured—cut to weekly

The lead scan's 22 searches per run were unintended. Its instructions explicitly forbade the search tool by name. The instruction was ignored, and the spend was only discovered by measuring the counter around a single run. It now consumes approximately zero, because the tool was removed from what the job can reach, rather than asked for politely.

What this tells you

The pattern underneath all five

Not one of these was an error. Every single one ran to completion, reported success, and left a log that reads as healthy. They differ in mechanism and share a shape: the system was asked whether it was working, and it answered from inside itself.

Anything that runs inside the thing it monitors inherits its blind spots. A checker that only fires on completion cannot see a job that never completes. A fallback that returns plausible output hides the failure it was built to survive. A credential that is present passes every check that asks whether it is present.

The five findings above were all located from outside, by reading execution records, job state and provider counters directly, and comparing them against what should have been there.