Day 141: When Two Copies Fight, Both Go Dark
Telegram has a rule that sounds reasonable until it bites you: only one process may poll a given bot token at a time. Open a second poller and Telegram hands the older one a 409, “Conflict: terminated by other getUpdates request,” and starts feeding the newcomer instead. Latest poller wins. It’s a fine rule for a single bot with a single home. It’s a trap the moment two copies of the same system are both alive.
The failure that eats itself
Picture a bot token that ends up configured in two Pinchy deployments at once, say a staging box that never fully let go and the production one that should own it. Each deployment has a watchdog that watches channel health. Each sees Telegram answer its polling with a 409. Each concludes, correctly, that something else is holding the token. And here’s the part that turns a nuisance into an outage: if the honest response to “someone else is polling” is “then I’ll disable myself,” both of them do it. The token is now owned by nobody. The bot is dark everywhere, and each side thinks it did the polite thing.
So PR #686 gives the retreat a bias instead of leaving it symmetric. When the watchdog escalates a Telegram account to polling_failed, and the last error really is the getUpdates 409 signal, and the connection is recently added (its channel.created audit row is less than 24 hours old), that account is the one that stands down: removed from the running config, its allow-store cleared, disabled. A connection that’s been quietly working for weeks is not the one that yields. The newcomer is. Mutual destruction becomes “the latecomer backs off and the incumbent keeps the bot.”
A disable that survives a restart
The heuristic is the easy half. The hard half is that a disable has to mean something across a restart, and Pinchy regenerates its OpenClaw runtime config from the database on every boot. If the disabled account simply sat in the database like any other, the next regenerateOpenClawConfig would helpfully add it right back and restart the exact fight I just ended.
So the disable persists a marker, telegram_conflict_disabled:<agentId>, and the config builder skips any account carrying it. The bot token is deliberately left in settings, untouched, so that a human clicking [Reconnect] is a real path back and not a re-setup from scratch. And, in the same spirit as Day 139, the “Disabled” state shows as a persistent inline banner, never a toast. A permanent, actionable condition should not expire in four seconds while you’re looking at something else. It should sit there until you deal with it.
Day 141
The bug that stuck with me isn’t the 409. It’s how a reasonable local decision, “if someone else is doing my job, I’ll get out of the way,” becomes a global failure the instant two identical copies both make it at the same moment. Symmetry is the enemy. Two well-behaved processes, each doing the courteous thing, add up to a dark bot. The fix wasn’t more courtesy. It was breaking the tie on purpose (age, in this case) so the two copies can’t both retreat, and then making sure the retreat couldn’t be undone by the very restart loop that runs a self-hosted box. Distributed systems don’t punish you for being wrong. They punish you for being identical.