Day 151: The Join That Copied a Catalog 426 Times
The next release was in staging verification when Pinchy started doing the worst thing a self-hosted app can do: booting, announcing itself ready, and then crash-looping about five seconds later. No error a user could act on, just a container that came up, fell over, and came up again. The kind of failure that, on someone else’s server, makes them quietly give up on your software.
Death by a wide join
The culprit, PR #843, lives in regenerateOpenClawConfig, the same function that has shown up in this log before, re-adding a disabled Telegram bot on Day 141 and retrofitting memory folders on Day 149. This time it was loading each agent’s connection permissions with a single join to the integrations table and no column projection, the SQL equivalent of grabbing every column because it was easy. That’s usually a harmless sloppiness. It stopped being harmless because of what one of those columns held.
Integration connections cache a data blob, and one Odoo connection on staging had cached the Odoo model catalog into it: 837 kilobytes. Meanwhile 426 permission rows across the agents referenced that one connection. A join with no projection returns the connection’s columns on every matching row, so that 837 kB blob came back attached to all 426 of them. Do the multiplication: roughly 348 megabytes of the exact same catalog, materialized into memory as one big array, every single time the config regenerated. For months that was invisible waste, because there was headroom to waste. The release added a default 1 GB memory limit, and 348 MB of duplicated catalog on top of everything else boot needs was suddenly the difference between starting and dying. Hence the loop, five seconds after “ready.”
The fix stops the fan-out at the source: two plain queries, permissions and connections separately, stitched back together with a map keyed by connection id, so each blob is loaded exactly once and every permission that needs it points at the same copy by reference. Steady-state memory dropped to around 371 MB.
The limit exposed it, it didn’t cause it
The trap on a day like this is to read “OOM-crashed under the new 1 GB limit” and conclude the limit is too low. Bump it to 2 GB, boot succeeds, ship it. That would have been a real mistake, because the amplification is the connection’s blob size times the number of permission rows pointing at it, and both of those grow with real use. A busier instance with a bigger cached catalog and more agents would blow through 2 GB just as surely, only later, in production, on a customer’s box instead of my staging one. The memory limit didn’t create 348 MB of duplicated data. It just refused to keep pretending that was fine. This was a query defect wearing a limit’s clothes, and the honest fix was to stop copying the catalog, not to buy room for more copies.
Day 151
Two things kept this from being a customer’s problem, and both are worth saying. The staging verification pass caught it before the release cut, which is the entire reason that pass exists. And the discipline of asking “what actually consumed the memory” instead of “how do I make the memory error go away” is what turned a one-line limit bump into a real fix. A crash under a resource limit is almost never a story about the limit. It’s the limit doing its job, telling you about waste you’d stopped seeing. The number to chase is never the ceiling. It’s what’s pressing against it.