Day 135: A Healthy Zombie
The worst kind of broken is the kind that looks fine. On staging this week I had a container that was, by every automated measure, perfectly healthy. HTTP was up. /api/health returned “ok.” Docker’s healthcheck was green. And every single chat session showed “Reconnecting to the agent…” forever, with a permanently red indicator, because the part that actually matters was dead and nothing was willing to say so.
A green healthcheck on a dead server
The trigger was mundane (PR #652). During a debug session someone had run a root docker exec, which left the device-identity file owned by root and unreadable by the app. So on the next boot the OpenClaw client constructor threw. Here’s the problem: the startup chain in server.ts was an app.prepare().then(...) with no terminal .catch. A throw in there doesn’t crash the process. It surfaces as an unhandledRejection, which Next.js dutifully logs and then swallows. The HTTP server had already started listening, so the container looked alive. It just had no OpenClaw wiring behind it: no connect, no status broadcaster, no watchdog. A zombie. Upright, warm, answering the door, nobody home.
The fix is one line of intent: startup.catch(exitOnStartupFailure). On a startup failure, log one loud, specific line naming what broke, then process.exit(1). Docker restarts the container, visibly, and if it can’t recover it enters a crash loop. A crash loop is ugly, but a crash loop is diagnosable. A zombie that passes its own healthcheck is not. I would rather have a server that admits it’s down than one that lies that it’s up. And the healthcheck itself learned to stop lying: it now reports actual OpenClaw gateway connectivity (PR #653), so “healthy” means the thing behind the door is home too.
The 200 that should have been a 503
The same instinct fixed a quieter version of the same sin (PR #658). A credentials route refreshes OAuth tokens on demand. When the Google OAuth app settings were missing during a refresh that was actually required, because the access token had expired, the code logged a warning and returned the stale, expired credentials with a cheerful HTTP 200. The email plugin then cached those doomed credentials for five minutes and failed later with confusing auth errors that pointed nowhere near the real cause. Returning 200 with something you know is broken is the API-layer version of the healthy zombie. Now that path throws a specific error and returns a 503 with no credentials, so the failure lands where the failure is, not five minutes downstream wearing a disguise.
Day 135
Every one of these was the system choosing to look fine over telling the truth, and every one of them cost more to debug precisely because it looked fine. This is the same value that runs under the whole project: an error’s job is to be legible, especially in a self-hosted deployment where the person debugging it is a customer’s ops team at a distance, with none of the context I have, reading only what the system chose to tell them. A loud crash respects that person. A green checkmark on a dead process wastes their afternoon and their trust. Fail loud. It’s the same lesson as Day 134, one layer down: the most useful thing an error can do is refuse to pretend.