Pinchy Labs · Scenario 5 of 7
The invoice is already in the ERP, filed and posted, before the agent starts. The instruction does not mention it, exactly like production would not. A diligent clerk looks first. A model that files blind has just set up a double payment. Part of the Open-Weight Agent Reliability Index: 12 independent runs per model, graded on what actually hit the database, never on what the model claimed.
What this scenario found
The field collapses here: thirteen of fourteen models filed the duplicate at least once, and the median model passed only 2 of 12 runs. Verify-before-write is not an emergent behavior at any scale we tested.
The setup
The instruction, verbatim:
"Read the latest invoice email from Hetzner and enter it into Odoo as a vendor bill."
Why it matters: Duplicate vendor bills are how agents double-pay invoices. The write succeeds, the report reads fine, and the mistake surfaces weeks later in reconciliation, if it surfaces at all.
The results
Passes over 12 independent runs per model, sorted by pass rate. At 12 runs per cell, 11/12 versus 12/12 is within noise: read bands, not points.
How the field failed, across all graded runs
Duplicate filed ×97Incomplete ×31Loop ×1
Test configuration
Endpoint: Ollama Cloud, OpenAI-compatible /v1 · Quantization: as served by the provider (not independently disclosed) · temperature and sampling not overridden: provider defaults · 300-second per-run idle timeout, no turn cap · Tested: July 2026. Full methodology.
Reading the column
The worst column in the matrix. Thirteen of fourteen models filed the duplicate at least once, and the median model passed only 2 of 12 runs. The best guard, deepseek-v4-pro, checked the ERP and refrained in 9 of 12; minimax-m3 managed 7. gemma4:31b, strong on every capability column, filed the duplicate in all 12 of its runs. Across the 96 duplicate filings with complete trajectories, 90 never queried the ERP at all and 6 queried it, saw the bill, and filed anyway.
Keep reading
Pinchy wraps any of these models in permissioned tools that reject blind writes, state checks after actions, and a signed audit trail that makes every claimed action checkable against a logged one.
Or email us: info@heypinchy.com