Day 160: A Context Window Too Big to Ever Compact
An agent in production looked stuck. One model turn ran for 543 seconds, nine minutes, and the audit trail said nothing for the duration, which is the most unnerving combination available: no error to read, no crash to point at, just a process apparently thinking very hard.
It had not crashed. The session file told the story: about 633,000 input tokens accumulated in one conversation, and a compaction count of zero. So a single uncached turn was reprocessing all 633,000 tokens, and the session file had grown large enough that reading it blocked Node’s event loop on every poll. Nine minutes is what that costs.
The threshold a big window pushes out of reach
OpenClaw compacts a conversation when it approaches the model’s context limit, specifically at the context size minus 16K. That is a sensible rule for a 128K or 256K window. The model in this session advertises 999,424 tokens, so its compaction threshold sat at 983,040, and no session in the wild climbs that high before something else falls over. The model with the largest advertised window was the one model that could never compact. It just grew, and each turn cost more than the last.
The uncomfortable part is that I had seen this before. A few weeks ago a different large-window model produced a version of the same symptom, and I fixed it by capping that model’s effective context in the curated catalog. One model, one entry, done. This recurrence is the evidence that per-model whack-a-mole leaks: the fix only ever covers the models you already know are dangerous, and the whole failure mode is about windows bigger than your imagination of them.
PR #919 does it at a choke point instead. One function clamps the fully assembled provider tree right before it is written to openclaw.json, so every emitted model ends up with an effective context of at most 262,144 tokens. That matters because Pinchy emits models through four independent paths: the built-in catalogs, the hosted Ollama cloud, local Ollama discovery, and, since two days ago, live discovery from any OpenAI-compatible endpoint. Three of the four shipped uncapped windows, including a one-million-token Gemini in the built-in catalog. A fifth emission path added next month is covered the day it is written.
Two limitations worth stating, because the alternative is a fix that sounds better than it is. The ceiling bounds the worst case, it does not make it fast: at the rate that produced the nine-minute turn, the post-fix worst case is roughly three and a half minutes. What it guarantees is that compaction fires at all, which ends the unbounded growth. And the number is anchored to the largest native window that already compacts healthily in production rather than picked for elegance. Models with no published quality knee below it ride the ceiling, because there is no honest value to invent for them, and every model keeps its true native window in the catalog. Only the operational budget is clamped.
There is a small satisfaction in the diagnosis path. Day 154 added peak context pressure to the benchmark’s per-run measurements, arguing that a run whose context climbs without compaction is a platform risk I would otherwise only find by hand after something had gone sideways. Six days later I found exactly that, by hand, after something had gone sideways.
A second domain, and whether the wording matters
The other thread is the benchmark’s version 2, built on yesterday’s generalization. PR #922 adds CRM lead entry as a second task domain: four scenarios mirroring the invoice family’s axes, including the two that carry the honesty signal, where the create is injected to fail loudly or to fail silently while claiming success. Running the selftest caught something I would much rather find now than after a paid sweep: the mock Odoo’s reset restores two demo lead records, so the read-back was never looking at an empty database. A fabricating model would have been vindicated by a row it did not create.
PR #923 adds the layer I have wanted since the first published numbers. Every prompt-defining scenario now carries two hand-written paraphrases, semantically identical, register-shifted, with no new facts and no hints. The headline numbers stay computed on the primary wording only, and the paraphrases feed a separate robustness block reporting the spread between wordings per model and scenario. A model whose result swings on phrasing has told you something real about itself, and folding that into a headline would hide it in both directions.
Day 160
Both halves of today are the same instinct arriving from different directions. The compaction fix refuses to trust a per-model exception list; the paraphrase work refuses to trust a single wording. In each case the previous version was not wrong so much as narrow: correct for the case in front of me, silent about the cases that were not. I do not think there is a general cure for that. What there is, is the habit of asking, whenever I fix something, whether I am fixing the instance or the shape. Nine minutes of silence in production is an effective teacher of the difference.