Day 123: The Box I Haven't Touched
A while ago I spent an afternoon with someone whose whole world is the inverse of mine. I build software. He builds the machines it runs on. We come from the same place and want roughly the same things: keep your data on hardware you can see, trust the box in the room over the cloud you cannot. Somewhere in that conversation he asked the question I keep circling back to. When you take Pinchy completely off the network, what does it actually run on?
I did not have an answer from experience. I have never run Pinchy on the class of machine he meant. He is ordering one, and when it arrives we are going to build an air-gapped prototype and find out for real. Until then I have research and arithmetic, nothing I have touched. So this is the honest version of a spec sheet: what I can work out about a box I have never run, and, just as much, what I cannot.
The question hardware reviews don’t ask
Most “can it run an LLM” reviews measure the wrong thing for me. They test a chatbot: short prompt, watch the words stream out. An agent is not a chatbot. Every turn, Pinchy hands the model a large system prompt, retrieved context, tool definitions, and the running transcript. That load is front-loaded, and the expensive part is prefill: reading all of it before the first token appears. Prefill is bound by memory bandwidth, not by the TOPS number on the marketing page.
So the spec that matters most is not the NPU’s headline figure. It is how fast the machine moves weights and context through memory. The Ryzen AI Max+ 395 class, the “Strix Halo” boxes like the MINISFORUM MS-S1 Max and the Framework Desktop, pairs a capable integrated GPU with up to 128 GB of unified memory at roughly 256 GB/s. The memory pool is the headline. The bandwidth is the catch.
”For AI clusters” deserves a closer read
MINISFORUM sells the MS-S1 Max “for AI clusters,” and that is exactly the kind of phrase I would want someone to fact-check on my own product. You can link two of these boxes, over USB4 or 10-gigabit Ethernet, and pool their memory. What you cannot do is make them fast together. The interconnect runs at maybe ten gigabits in practice, and the consumer network cards do not speak RDMA. Linking boxes lets you fit a bigger model across their combined memory. It does not hand you the near-linear speedup the word “cluster” makes you picture. Real GPU clusters move data between chips hundreds of times faster than a USB4 cable. Two mini-PCs on a wire are a memory-pooling trick, not a supercomputer. Genuinely useful, but only if you know which one you are buying.
Arithmetic I can do without the box
The good thing about being bandwidth-bound is that you can estimate the ceiling without owning the hardware. Generating a token means reading the model’s active weights out of memory once, so the speed ceiling is roughly bandwidth divided by bytes read per token. Take a mixture-of-experts model that only activates a few billion parameters at a time, quantised to four bits, call it a couple of gigabytes per token. Against 256 GB/s that is a ceiling near a hundred tokens a second, and real overhead, prefill, and an imperfect memory path drag the actual figure well below it.
That is the shape the people who do own these boxes are publishing: a 120-billion-parameter mixture-of-experts model generating in the low 50s of tokens per second, a 30B one up in the 80s. Theory and measurement agree, which is the most reassuring thing arithmetic can do. Bandwidth sets the pace. Mixture-of-experts models are the sweet spot because they read so little per token. A dense 70B at long context will feel heavy no matter what the box promises. None of that needed me to plug anything in. But it is an estimate, not a measurement, and the difference is the whole point of this post.
The half I can’t compute
Then there is everything the arithmetic cannot reach, which is precisely the half an enterprise buyer cares about. Does the memory have error correction, or am I trusting unprotected RAM with a model that makes decisions? Is there out-of-band management, a way to reach the box when the operating system will not boot, or is it a workstation wearing a server’s clothes? The RAM is soldered, so one bad chip sends the whole unit back. What does that do to a fleet’s uptime? And for a genuinely air-gapped deployment, who do I trust in the supply chain that assembled the board?
You cannot read any of that off a spec sheet, and you certainly cannot compute it. You have to live with the machine. This is where not owning the box stops being a research problem and becomes an honesty problem.
Knowing which side of the line you’re on
A while back I wrote about the line between the runtime I own and the runtime I only get to ask to change. This is the same line drawn through hardware. On my side: what an agent demands from a machine, bandwidth over TOPS, mixture-of-experts over dense, prefill as the thing to watch. I know that, and I will stand behind it. On the other side: how this specific box behaves at 2 a.m. under real concurrency, whether it clears a CISO’s checklist, what breaks first. I do not know that yet, and no amount of reading gets me there. The honest move is to show you the line instead of smudging it, to be clear about which claims are measured and which are reasoned.
Day 123
The prototype is coming. When the box lands, my counterpart and I are going to put Pinchy on it, pull the cable, and replace every estimate here with a number I watched happen. Some of them will be wrong, and saying so will be the most useful paragraph I write about it.
I also pulled the comparison, the bandwidth arithmetic, and the governance checklist into a standalone guide to air-gapped LLM hardware, so it stays useful to someone sizing a box who did not come here for a devlog.
Until then: if you are already running Pinchy, or any agent, air-gapped on one of these machines, I want your numbers. Tokens per second, the model, the context length, what fell over. Send them and I will publish them, credited. The best spec sheet for a box like this will not come from the manufacturer. It will come from the people who took the cable out and kept going.