Day 150: Building In the Ways I Could Be Wrong
The reliability index grades open-weight models on whether they stay honest when a tool fails. Yesterday was about the grader being wrong. Today is the harder, less technical question: why should anyone believe the numbers at all? Benchmarks have a dismal track record, and they don’t usually fail by getting the arithmetic wrong. They fail in ways that happen to flatter whoever published them. So I spent the day building the specific ways I could be wrong, or be accused of cheating, into the benchmark itself, before putting it in front of anyone. This is the same move as auditing my own marketing for lies, pointed at a number instead of a landing page.
The four ways a benchmark lies without meaning to
Contamination first. The scenarios and trajectories live in a public repository. The moment they’re crawled into a model’s training data, a high score stops measuring reliability and starts measuring memorization, and you cannot tell the difference after the fact. So PR #805 embeds a canary, a unique marker in every scenario and every published file, the same trick BIG-bench used. It doesn’t prevent contamination. It makes future contamination provable, which is the honest version of a guarantee you can’t actually enforce.
Solvability second. A benchmark is only fair if every task can actually be won. SWE-bench Verified had to throw out more than two-thirds of its tasks as broken or impossible, and with only seven scenarios here, one unwinnable task would skew everything. PR #808 gives each scenario an oracle solution, a known-good trajectory that CI replays through the real graders and asserts passes, plus a mirror failure that must be rejected. If a scenario ever becomes unsolvable, the build goes red before anyone runs a model against it.
Funding third, and this one costs nothing technical and everything in discipline. PR #814 states outright who pays: the runs are on an ordinary list-price Ollama Cloud subscription, no vendor funds it, sponsors it, or discounts it, and there’s no commercial relationship in either direction. The point is the timing. FrontierMath’s damage came from disclosing its funding late; a disclosure after launch reads as a confession, the same one before launch reads as a standard. So it lands before the benchmark goes anywhere.
The claim I most wanted to overstate
The fourth is the one I had to talk myself into. It’s tempting to publish a clean ranking, model A beats model B beats model C, because a ranking is what people want. PR #809 ran the actual statistics on the differences, and the result is humbling: of 91 model pairs, 76 are statistically tied. Only 15 genuinely separate. At twelve runs per cell, this benchmark can tell a clearly-weak model from a clearly-strong one and very little in between, and the honest thing is to say exactly that on the page rather than draw fifteen decimal places of false precision. Alongside it, PR #806 publishes the pass-to-the-k reliability curve, because a model that passes ninety percent of the time individually only strings eight clean runs together about fifty-seven percent of the time, and one model drops from a respectable 0.75 single-run to 0.018 across eight. If you’re going to hand an agent repeated real work, the single-run number flatters it and the repeated-run number is the truth.
Day 150
Every one of these makes the benchmark look less impressive and more trustworthy, which is the trade I keep choosing across this whole product. A canary admits the questions can leak. An oracle admits a task might be broken. A funding note admits the incentive exists. A statistical tie admits I can’t rank models I’d love to rank. None of that is required. A less careful benchmark with a cleaner-looking leaderboard would get more attention. But the entire reason to publish reliability numbers for self-hosted models is that someone might bet their company’s bookkeeping on them, and that reader is exactly the kind who checks. Credibility isn’t a tone you write in. It’s the ways you could be wrong, found before your critics find them, and printed on the page next to the number.