Day 112: Watchdogs for Silent Failures
A crash is, in its own way, a courtesy. It tells you something happened, it usually tells you where, and it gives you a stack trace to argue with. The failures I spent today on do none of that. A Telegram channel that just stops receiving doesn’t crash; it goes quiet. A run that wedges before the first token arrives doesn’t error; it hangs. And from the other side of the screen, both look identical: the user typed something, and the agent is ignoring them. That’s the worst thing an assistant can do, and it’s the one failure mode that ships with no alarm attached. Today was three fixes, and they’re really one idea wearing three costumes: turn silence into a signal.
The resume that crashed on its own memory (PR #470)
The first one is the loudest of the three, which is why it’s the easiest. When a stream resumes (a browser reconnecting to a run already in flight, the machinery I built back on Day 99), it replays the chunks it missed so the reconnecting client catches up. The bug: resuming could replay a message id that already existed on the client, and that collision crashed the stream. So the very mechanism meant to make a dropped connection survivable could, on the unlucky path, take the whole stream down with it. The fix dedupes on resume so a message id that’s already present is recognised as a replay rather than treated as a new arrival that must be created twice.
I want to be honest about where this one came from, because it’s a familiar shape. The resume feature was the architecturally satisfying win of Day 99, the thing I was most pleased with that week. And it carried a crash that only surfaced on the resume-after-reconnect path, which is exactly the path that’s hard to hit in normal testing and easy to hit in production, where tunnels and sleeping laptops and flaky hotel wifi are the baseline. “I built it” and “it works in production” are different claims. I keep relearning that the bug is rarely in the code I’m worried about. It’s in the code I’ve already congratulated myself for.
The channel that went quiet (PR #473)
This is the one that bothered me most, because the failure is invisible by construction. Telegram’s getUpdates is how a bot polls for new messages. If two pollers ever fight over the same bot (two instances, a leftover process, a restart that didn’t fully let go), Telegram answers the loser with a 409 Conflict. And then nothing. The channel doesn’t error in any way the user can see. It simply stops receiving. Messages go into Telegram, Telegram has decided who owns the poll, and the side that lost the fight sits there politely receiving 409s that no one is looking at.
From the operator’s chair, this is maddening, because the system is technically working: every component is up, no exception is thrown, the logs aren’t red. It’s just deaf. The fix is a channel-health watchdog that watches for exactly the 409-conflict condition and surfaces channel health as a first-class signal instead of letting the channel go silent. The point isn’t that the watchdog repairs the conflict; it’s that a deaf channel now says it’s deaf. The failure becomes attributable: you can see which channel, that it’s a poller conflict, and that the silence is a known state rather than a mystery. A 409 you can read is a different animal from a 409 that just makes your bot stop existing.
There’s a governance angle here that matters for the kind of teams Pinchy is for. If a channel can fail silently, then “is the agent still reachable?” is a question nobody can answer with confidence, and that uncertainty is its own liability. Making channel health visible is the same instinct that runs through the audit trail: a system you can be accountable for is one that tells you what state it’s in, especially the bad states.
The run that wedged before it started (PR #474)
The third one closes the gap on the other end of the pipe. On Day 99 I added a server-side watchdog that tears down runs past a hard time cap and audits runs that finish after the browser has left, but that machinery keyed off a run that had at least started streaming. A run that wedges with no first chunk ever arriving fell through: nothing to tear down, because from the server’s point of view nothing had begun. So the stream would just sit there, the spinner spinning, no first token, no error, no end.
PR #474, labelled B-1 in my notes, adds a server-side first-chunk watchdog. If a run goes too long without producing its first chunk, the watchdog times the silent stream out and reports it instead of letting it hang. It’s the missing bookend to Day 99’s work: that watchdog caught runs that hung after producing output; this one catches runs that never produced any. The shape is the same as the other two fixes: a quiet hang gets a deadline, and crossing the deadline produces a visible, attributable event rather than an indefinite wait. The honest version of “I don’t know what’s happening” is still infinitely better than a spinner pretending everything is fine.
Day 112
Two days ago, Day 110 was about honesty as a system property: the idea that the most useful thing software can do under stress is tell you the truth about its own state. Today is the operational reading of that same principle. A silent failure isn’t more honest than a loud one; it’s less. It withholds the one thing the operator needs, which is the knowledge that something is wrong at all. Each of these three watchdogs does the unglamorous work of converting an absence into a presence: a crash where there was a quiet corruption, a health event where there was a deaf channel, a timeout where there was an endless wait.
None of this is demonstrable in the way a feature is. There’s no screenshot for “your Telegram channel told you it had stopped before your customer did.” The whole payoff is, once again, an absence that becomes a signal, and the uncomfortable lesson of a hundred and twelve days is that the failures worth chasing hardest are the ones that never raised their hand. The crash will always get fixed, because the crash complains. It’s the silence you have to go looking for.