Day 155: Everything Was Fine Except the Part That Mattered
Someone upgraded their self-hosted Pinchy the way people actually upgrade things: docker compose pull, restart, keep the compose file that has been sitting in the directory since installation day. That is a completely reasonable thing to do, and it broke their instance in a way that took three separate fixes to make visible.
One missing mount, and a config file frozen in time
The compose file they kept predated the openclaw-secrets volume, so the new images came up without that mount. Pinchy runs as uid 999 and cannot create a directory at the container root, so writing the secrets file threw EACCES, config regeneration aborted mid-flight, and openclaw.json stayed exactly as it had been before the upgrade. From that moment on, every provider they added and every agent they created went into the database and never reached OpenClaw, which was still faithfully running a configuration from the previous version. The two halves of the system were telling different stories, and the half the user could see was the optimistic one.
PR #891 fixes the diagnosis: a boot preflight checks that the secrets directory can be created and written before anything needs it, and the bare EACCES is replaced, at the single choke point every regeneration flows through, with a message naming the cause and the recovery. It logs loudly at boot and deliberately does not block boot, because an instance that refuses to start is an instance the operator cannot log into to fix their compose file.
Three layers, three soothing lies
What makes this worth writing down is not the missing mount. It is that the failure passed three surfaces on its way to the user, and each one had a way to stay quiet.
The health endpoint stayed green. Readiness has to fire unconditionally, because OpenClaw waits on Pinchy being healthy before it starts, so gating it on a successful regeneration would mean a broken volume prevents the runtime from ever coming up, including on fresh installs. The answer was not a stricter readiness check but a separate signal: configRegeneration: { ok, error, at }, carrying the actionable message from the preflight and self-healing on the next success. A green light that cannot go red is not a safety feature, it is decoration.
Saving a provider returned an HTTP 500 for a change that had already committed. Both write routes wrote the row and then regenerated unguarded, so on this box every save threw after the write succeeded: the user saw a hard error, refreshed, and found the provider sitting there. PR #888 makes the runtime apply best-effort. The route returns success with a warning, the client raises a warning toast, and the audit detail records runtimeApplied: true|false so the trail distinguishes a change that was saved from one that was saved and applied. “Failed to apply” must not be reported as “failed to save,” because those send a person looking in completely different places.
And the chat said nothing useful. The logs on that box carried a 410 for a retired model and unknown agent id, while the chat window showed “Something went wrong. Please try again.” and the error table held zero rows. The reason has an uncomfortable symmetry with Day 152: there were two ways a run failure could arrive, and only one was handled richly. An error yielded as a chunk on the run stream got classified, audited, persisted and turned into a banner naming the dead model. An error thrown as an exception from the same generator hit a catch that sanitized it into the generic bubble. PR #889 routes both through one terminal-failure sink and adds a model_retired class, so a retirement stops falling through to unknown and reaches the durable banner that already knows how to name the fix.
Day 155
A system rarely fails silently in one place. It fails once, and then a chain of individually defensible decisions each declines to raise its voice: readiness must not be gated, exception text must not be leaked, a fallback must be graceful. Every one of those is correct on its own, and together they produced a box that reported itself healthy while quietly running last week’s configuration. The rule I am taking: a component that cannot apply a change owes the user a louder signal than one that cannot save it, because a failed save is obvious within seconds, and a change that saves but never applies can go unnoticed for a version and a half.