Day 106: Knowing What the Model Can See
A model can be blind. I knew this in the abstract (some models read images, some only read text), but for a hundred and five days Pinchy treated that distinction as someone else’s problem. You’d attach a screenshot, hit send, and find out the hard way. The request would error, or the image would vanish somewhere in the pipeline and the model would answer as if you’d sent nothing. Either way you’d already spent the turn. The agent had committed to a task it couldn’t perceive, and only discovered the gap in the middle of doing it.
Today’s fix, PR #343, is the least glamorous thing I’ve shipped in a while: a table. A list of which models can do what. But it closes the gap between an agent that fails at the door (politely, before you’ve committed anything) and one that fails somewhere down the hall, mysteriously, after it’s too late to take it back.
The hope I was running on
Here’s what I’d been doing, which I can now name as a mistake. When a template wanted vision (a document-analyzer, say, that’s supposed to look at a scanned invoice), I picked its model by reputation. I knew Claude could see, and Gemini could see, and I wired the picker to reach for those. That worked right up until it didn’t. A model gets renamed. A provider ships a text-only variant under a name that looks like a vision one. The user has only configured a local Ollama model that happens to be text-only, and the template still confidently selects it because nothing in the system actually checks. I was encoding capability as folklore, things I happened to remember about each model, instead of as a fact the system could read.
The attachment path was the same hope wearing different clothes. Nothing stood between “user attaches an image” and “request goes to a model that can’t process it.” The failure mode wasn’t a missing feature; it was a missing question. Pinchy never asked, before sending: can the thing I’m about to send this to actually see it?
A table of what’s true
So now there’s a models table, and it records per-model capabilities: vision, long-context, tool calling. It’s seeded automatically at every boot from a built-in catalog covering Anthropic, OpenAI, Google, and Ollama Cloud: the providers Pinchy knows how to talk to. For local models, where I can’t ship a catalog because I don’t know what you’ve pulled, it reads capabilities live from the Ollama API. The catalog is the part I can know in advance; the live read is the part only your machine knows.
Re-seeding at every boot matters more than it sounds. Capability isn’t a one-time fact I write down once and trust forever. Providers change their lineups, and a stale table is just folklore with a database schema. Reading it fresh on each start is how the table stays honest. (Day 105’s lesson is still ringing in my ears here: a mechanism you built and then assumed keeps working is exactly the kind that quietly stops. A table that re-seeds is a table that can’t drift into a lie I’d have to discover in production.)
With that table in place, two things stop being guesses.
The first is template model selection. When a template declares it needs vision, the picker now selects the first configured model that actually meets the required capabilities, read from the table. Not the one I remembered being good at it, but the one the record says can do the job. If your only vision-capable configured model is some specific Gemini, that’s what a document-analyzer gets, because that’s what’s true, not because I hard-coded a preference six weeks ago.
The second is a pre-send check, and this is the one that changes the texture of using Pinchy day to day. If you attach an image and the agent’s current model can’t see, Pinchy tells you before sending. It doesn’t quietly drop the image and let the model bluff its way through a description of nothing. It doesn’t let you spend the turn and then hand you an error. It stops at the door and offers to switch you to a model that can actually do what you’re asking. You confirm, it switches, the image goes to something with eyes.
The one thing I almost broke
The first version of this check was too eager. I’d written it to block any attachment going to a model that lacked vision, and that’s wrong for PDFs. PDFs in Pinchy don’t ride the image path at all; they go through a dedicated PDF tool, and that tool picks its own model separately. So a perfectly normal “here’s a PDF, summarize it” against a text-only chat model is fine: the chat model never sees the PDF as an image, the PDF tool handles it with whatever model it selected. My over-broad check would have warned the user about a problem that didn’t exist, told them to switch models they didn’t need to switch, and taught them to distrust the warning. A false alarm at the door is its own kind of failure: it trains people to push past the guard. So PDFs are never blocked. The check is narrowed to the case it’s actually for: an image going to a model that can’t see images.
That correction is the whole shape of the feature in miniature. The point was never to add friction; it was to put the friction in the one place where it saves you from a worse surprise later, and nowhere else.
Day 106
What I shipped today is bookkeeping, and I want to resist dressing it up as more than that. There’s no clever inference here, no model deciding anything. There’s a table that says what’s true, read fresh on every boot, consulted before Pinchy commits you to a turn. The whole improvement is that capability stopped being something I hoped about and started being something the system could look up.
The pattern underneath it is one I keep relearning in different costumes. The expensive failures aren’t the loud ones; they’re the ones that let you proceed. An agent that accepts whatever you hand it and discovers the mismatch halfway through has cost you the task and taught you nothing about why. For an enterprise tool where every action is supposed to be legible and accountable, failing clearly and early, with a way out, isn’t a nicety. It’s the contract. A model can be blind, and the least Pinchy can do is know it before it asks the model to look.