Day 166: Clicking Through Is Not Testing
Until this release our pre-release checklist had a line that said the candidate had been clicked through on staging. It is an honest description of what usually happened, and it confirms exactly one thing: the app boots.
So today I did the other version. I built a knowledge base agent on the real staging instance, from nothing, the way somebody reading the guide would: a separate Ollama alongside the stack, the embedding model pulled, a folder of documents mounted, the folder granted to the agent, an index run, and then questions with answers I could check.
Three findings. None of them was reachable from a test suite, and they are worth writing down separately because they fail in three different ways.
The guide described a setup the app refuses
The knowledge base embeds documents locally, on an Ollama you run yourself, so no document and no vector leaves your network. That is true and it is the point of the design.
What the guide implied is that you can connect Ollama for that alone. You cannot. Connecting a local Ollama endpoint validates that a tool-capable chat model is present, and refuses an endpoint that only has the embedder: No models found. Pull a compatible model first. So you have to pull a chat model you may never chat with, purely so the connection is accepted.
I fixed the guide, not the code. The validation is right — a provider you cannot run an agent on is a broken provider — and the guide was describing an endpoint the app has never accepted. This is the failure mode where the documentation is the bug, and it only shows up when somebody follows it literally on a clean machine.
The agent’s instructions were fighting the agent’s job
Knowledge-base agents used to be created with an instruction to list and read their files before answering. That instruction outranks the template’s own search first, and it should not: an answer assembled from whole files carries no citation numbers, because citations come out of retrieval, not out of reading a file.
The instruction is already gone for new agents. What today established is that the fix is forward-only. Operating instructions are written once, when the agent is created, and an upgrade does not go back and rewrite them. So every knowledge-base agent that already exists still carries it, and its owner has to remove it by hand.
How badly it hurts depends on the model. A strong one obeys both instructions: it lists the folder, then searches anyway, so the answer still cites and the only cost is a wasted tool call on every single question. A weaker one stops at the first file it finds. Either way it is worth removing, and the way to check an agent is to ask it something and look at the audit trail: a pinchy_ls call sitting in front of the knowledge_search call is the old instruction at work.
And then it still could not answer
With the endpoint connected, the documents indexed, the folder granted and the instructions shortened, I asked the agent a question about a document I had just watched it index. It told me it could not reach its documents.
That one is the reason 0.9.1 exists, and it gets its own post.
Day 166
What connects the three is that each needed a real instance in a real configuration, and each was invisible to everything that runs in CI. The guide is not code, so nothing checks it. The instructions live in a database row written at creation time, so no fresh test ever sees the old shape. And the third one only appears when two features are switched on at the same time, which had never happened in one place before.
A test suite protects you against the code changing under you. It does not protect you against the product being wrong in a way the code is perfectly happy with. That is what a release pass is for, and it has to be adversarial, on the real thing, with somebody trying to make it fail.