Day 154: A Model That Fails Cheaply Isn't Cheap
For two weeks the reliability index could answer one question well: when a tool fails, does the model tell the truth or fabricate a success? That’s the question I care most about. But it’s not the only question a person picking a model has to answer, and the one it was silent on is money. A model that’s perfectly honest and costs ten times another to do the same job is a real trade-off, and the benchmark had nothing to say about it. Today it does.
Joining a run to its cost, exactly
PR #839 captures per-run token usage, and the interesting part is how it attributes cost to the right run without guessing. Each eval run mints a fresh chat, which means it occupies a unique session key in OpenClaw, which means every usage record stamped with that session key belongs to exactly that run and no other. So the cost of a run is just the sum of the usage rows under its session key: the whole tool loop, every retry, all of it, with no time-window fuzziness and no risk of grabbing a neighbor’s tokens. It also records peak context pressure, the high-water mark of how full the context window got, which is the read-side of the Piper incident from Day 147: a run whose context climbs and climbs without compaction kicking in is a platform risk, and now it’s visible as a number instead of something I only find by hand after a bookkeeping agent has already gone sideways.
The collector is deliberately best-effort. If the database hiccups while it’s reading usage, the run gets an undefined cost rather than the whole sweep aborting, because a fifteen-hour benchmark run that dies on a transient blip in hour twelve is its own kind of disaster, and a missing cost on one run is a far smaller loss than the eleven hours before it.
The median that refuses to reward giving up
Here’s the measurement decision I most want to remember. Cost per task is reported as a median over completed runs only, not over all runs. It would be easier to average every run’s tokens and print that. It would also be a lie of exactly the shape Day 148 was about. A model that gives up on a hard task after two turns burns almost nothing. Average that near-zero in and the quitter looks like a bargain, when what it actually did was fail cheaply, which is not a bargain at all, it’s the worst outcome dressed in the smallest number. You paid a little and got nothing, and then a human has to do the whole task anyway. So the cost that gets published is the cost of the runs that actually finished the job, because that’s the only cost comparison that means anything: among the models that can do the task, which does it for less. A model that can’t do it doesn’t get a cheap-price sticker for not trying.
For the self-hosted, open-weight models the index focuses on, some are billed by subscription rather than per token, so their dollar figure comes as a labeled range rather than a false-precise number, another case of printing the honest uncertainty instead of a clean lie.
Day 154
That closes a two-week stretch that turned out to be almost entirely about one thing: what it takes to make a number trustworthy. The grader that credited crashes as honesty. The benchmark that had to disclose its own funding and admit its models were statistically tied. The pseudonymization that protected two rows out of 1221. And now a cost metric that refuses to let failing cheaply masquerade as being cheap. The through-line is that a measurement is a claim, and a claim is only as honest as the failure modes you went looking for before you published it. Reliability and cost are the two numbers someone will actually bet their company’s work on. The least I can do is make sure that when they read a small one, it isn’t small because the model quit.