Day 158: Two Lists I Wrote When I Had One Example
Two refactors landed today, in parts of the system that have nothing to do with each other, and they turned out to be the same mistake made twice. In both cases I had written a list at a moment when I had exactly one example of the thing, and the list quietly became the definition.
The benchmark that could only grade invoices
The reliability index measures one thing well: when a tool fails, does the model report the failure or fabricate a success. It measures it on invoice processing, because invoice processing is what I built first, and over time the domain stopped being a parameter and became furniture. The read-back that verifies what actually landed in Odoo said account.move. The helpers that inject a failure or a fake success said account.move. The grader’s phrase lists, the calibrated wordings that tell a claim of success from an honest report of failure, were one flat set tuned on invoice language.
PR #871 pulls the domain out into a parameter at each of those three points: scenarios declare which models to read back, the injection helpers take a target model, and the phrase lists become per-domain sets with the invoice slot pointing at the existing calibrated arrays, untouched. Nothing about the published numbers may move, and “may not move” is a weak promise unless something enforces it, so the enforcement is a fingerprint over every published result that CI checks on every commit.
That is the part worth taking rather than the code. A refactor of a measurement instrument is only trustworthy if the instrument’s outputs are pinned by something other than the person doing the refactor. Otherwise “I was careful” is the whole argument, and I have already written several posts this month about how far careful gets me.
The provider list that was a type
The other list was in the product. Pinchy knew five providers, and it knew them as a fixed union of names in the type system, which meant the type system was enforcing a business decision made months ago on the basis of what I happened to have accounts for.
PR #905 adds a generic OpenAI-compatible provider type, of which you can configure as many instances as you like: a display name, a base URL entered verbatim including the /v1, an API key. Models are discovered by asking the endpoint for its model list, with manual entry as a fallback, and capabilities come from a vendored snapshot with conservative defaults for ids nobody has heard of.
What this unlocks is the thing customers here keep asking for. Swisscom, IONOS, OVHcloud, Scaleway and T-Systems all serve OpenAI-compatible endpoints, and none of them was ever going to be in a hardcoded five. Neither was the vLLM instance somebody runs in their own rack, or a gateway in front of a mix of both. For a platform whose whole argument is that you should run your agents on infrastructure you control, having the reachable providers be a list I maintain was a contradiction sitting in plain sight.
The construction detail worth recording is that the five built-ins were not touched. The fixed union and its storage stay as they were; custom instances live in a new table with an additive migration that is empty on upgrade, and every place that enumerates providers walks both. Model list, config emission, secrets, default provider, agent migration, model resolution, the guard that stops you deleting your last provider. It is more call sites than a grand unification would need, and it is the version where an upgrade cannot break a working instance. Keys are encrypted at rest and only ever emitted into openclaw.json as a reference, with a scanner test that fails the build if a plaintext key ever appears there.
Day 158
Both lists were written honestly. When there is one domain, naming it is clarity rather than sloppiness, and when there are five providers, a union type is genuinely the safer design. The failure is not writing the list. It is that a list never announces the day it becomes a constraint. Nobody files a bug titled “your type system has an opinion about European cloud providers.” You find out because a customer asks for something reasonable and the answer takes three weeks, or because a second task domain turns out to require touching the grader, the injector and the read-back before it can exist at all. The cost is not paid where the list is. It is paid in the shape of every question you are then slow to say yes to.