Day 136: Debugging a Box You Can't SSH Into
The healthy zombie from a few days ago has been rattling around in my head, not because of the bug itself, but because of how I found it. I found it because it was my staging box and I could get into it. Now play the same incident on a customer’s hardware, in their data center, behind their firewall, and ask the real question: how does that get fixed?
The privilege I designed away
The honest answer is that I can’t get in, and that’s not a gap, it’s the point. A self-hosted Pinchy runs on the customer’s infrastructure precisely so that I don’t have a key to it. The whole pitch is that their data and their agents live somewhere I can’t reach. Which means the debugging move I used all week, log into the box and look, is a move I structurally do not get to make in the deployment that matters most. SaaS founders never have this problem: it’s all their box. Self-hosting trades that away on purpose, and then hands you the bill when something breaks two thousand kilometers away.
So the diagnostics have to travel, because I can’t. Instead of me reaching into the machine, the machine has to be able to hand out a faithful, bounded description of what went wrong, and the customer decides whether to send it.
Make the box explain itself
That’s the thread running through this week’s less flashy work: a diagnostics bundle that a user can export themselves. It captures the pieces I’d actually reach for, the agent’s configuration, which model and tools and prompt it was running, and the trajectory of what happened. The part I like most is that it fills the gaps in that trajectory from Pinchy’s own audit log: where the runtime’s capture truncates a deeply nested tool-call argument, the bundle swaps in the fuller copy Pinchy already recorded, and stamps each call so a reader knows which came from where. The audit trail stops being only a compliance artifact and becomes a debugging one. Same record, second job.
The design constraint that makes it actually usable is that it degrades gracefully and stays inside the user’s own scope. A bundle you can’t export because one chat is missing is useless in the exact moment you need it, and a bundle that quietly scoops up things the user didn’t mean to share is worse than useless, it’s a breach of the promise the whole product is built on. So it exports what it can, tells you honestly what it couldn’t, and never reaches past what the user is allowed to see.
Day 136
This is the same restraint as the air-gap box I still haven’t touched, viewed from the support side instead of the security side. When you deliberately give up the ability to reach into the running system, you owe the person who runs it a way to see what you would have seen. Observability you can export is the self-hosted substitute for a login you’re never going to get, and it’s a substitute I’d rather have anyway, because it puts the customer in control of exactly how much they tell me. The next stretch of this work is making that bundle richer and easier to produce, so “it’s broken” turns into “here’s precisely how” without anyone handing over a key.