Day 108: Making the Tests Tell the Truth
A flaky test is a message you keep ignoring. It fails, you rerun it, it passes, you move on. Every rerun is a small vote for the theory that the test is unreliable rather than the code. Today I stopped voting that way, because the test I’d been mentally filing under “noisy” turned out to be the most honest thing in the suite. It wasn’t flagging itself. It was flagging a race that could hit production after any OpenClaw restart.
The flake that was telling the truth
The dispatch end-to-end test fails intermittently. Most of the time it passes; sometimes it doesn’t, with no obvious pattern. The convenient story is that E2E is just flaky: there’s a real machine, a real OpenClaw process, a real WebSocket, and any of those can hiccup. That story is comforting because it ends with “rerun it,” and a green rerun feels like a resolution.
It isn’t one. When I actually traced the failing path, the flake had a precise cause. The test restarts OpenClaw, exactly what happens in production every time config changes and the gateway has to come back up. After that restart, two things need to happen before an agent can take a message: the new config has to be pushed to the gateway, and the gateway has to signal that the agent is ready. Nothing ordered those two events. So there was a window where the config push and the readiness signal raced each other, and a message sent into that window arrived at an agent that wasn’t ready yet. In the test, that showed up as an intermittent failure. In production, it would show up as a message quietly going nowhere right after a restart.
That reframes the whole thing. The test wasn’t unreliable: it was reproducing, at random, a real ordering bug that the rest of the suite was too fast or too lucky to expose. Reruns weren’t noise-cancelling. They were evidence-destroying.
Order the signal, don’t retry the symptom (PR #464)
The wrong fix here is seductive precisely because it works on the dashboard. Wrap the send in a retry, give it a couple of attempts with a short backoff, and the E2E goes green and stays green. CI is happy. The problem is what that retry hides: in production the same race is still there, and the retry I’d have added to the test isn’t running in front of a user’s first message after a restart. A green test bought with retries would have been a worse outcome than a red one, because it would have told me the bug was gone when it had only gone invisible.
So PR #464 fixes it at the root: a deterministic agent-readiness gate, and a config push that reliably lands across an OpenClaw restart. The send doesn’t fire into a maybe-ready gateway and hope; it waits on a readiness signal that’s actually ordered after the config push. The two events that were racing now have a defined sequence. The dispatch E2E stops flaking not because it retries until it gets lucky, but because the thing it was catching at random can no longer happen, in the test or in production. The flake disappears as a side effect of the bug disappearing, which is the only disappearance I trust.
This is the whole case for an end-to-end test: it exercises the actual ordering of actual processes, not a hand-built imitation of how I assume they behave. A unit test that mocked the readiness signal would have happily returned “ready” on cue and never seen the race. The race only lives in the seam between two real components, which is exactly the seam an E2E is for. The cost of that fidelity is that the test sometimes fails for reasons that look like infrastructure, and the discipline is to assume it’s load-bearing until proven otherwise.
Totals, not just presence (PR #462)
The second landing is a quieter version of the same lesson. Token-usage accounting had tests, but they were checking that usage numbers were present: that a total existed and looked plausible. PR #462 implements the Tier-2 token-total tests from #426, which assert on the totals themselves, not merely that a field showed up. A test that checks presence is a test that passes whether the number is right or wrong; it confirms the plumbing without confirming the arithmetic. Asserting on the totals means the test now fails if the accounting drifts, which is the entire point of having it. Presence checks are how a suite stays green while the thing it’s supposed to protect quietly goes wrong.
The catalog that drifted from reality (PR #456)
The third landing is a drift of a different kind. The Ollama Cloud model catalog Pinchy ships had fallen out of sync with what Ollama Cloud actually serves over its live API: the list described a world that no longer quite matched the one a user’s agent would hit. PR #456 reconciles the catalog against the live API, so the models Pinchy offers are the models that actually answer. It’s a small, unglamorous reconciliation, and it rhymes with the day’s theme: a description that’s stopped matching the territory is a quiet lie, whether it’s a test asserting nothing real or a catalog listing models that have moved on. Both look fine until someone relies on them.
Day 108
The through-line of the day is that I spent it making three artefacts stop lying. A test that failed at random and got rerun into silence. A token assertion that checked for a number’s existence but not its truth. A model catalog that had drifted from the live API. None of them were loud about being wrong. That’s the trap. A flaky test in particular is the most ignorable bug report in the world, because it files itself, withdraws itself on the next run, and never raises its voice. The uncomfortable part is that the dispatch flake had been telling me about a production race for a while, and the only reason I heard it today is that I stopped assuming it was noise. “I built it” and “it works in production” are different claims; a test that fails at random is sometimes the only thing standing between the two.