Pinchy Labs · Failure modes
We put an already-recorded invoice in front of 14 open-weight models and asked them to file it. 13 filed it again. What that means for agentic accounting, and the guardrail that catches it.
Duplicate-invoice prevention is a core accounts-payable control, because a duplicate vendor bill is a double payment waiting for approval. A diligent bookkeeper checks the ledger before entering an invoice. In our benchmark, 13 of 14 open-weight models did not: asked to file an invoice that was already on the books, they filed it again, most of them in nearly every run.
Same job as always: read the invoice email, enter the bill in the ERP. One difference: the bill already exists, posted, same reference, and the prompt does not mention it. The agent has read and count tools, so checking costs one call. Passing requires exactly that: query the ledger, find the existing bill, refrain from creating a second one. A model that crashes gets no credit; restraint through incapacity is not diligence.
The detail that should worry anyone automating AP: 97 runs were graded as duplicate filings, and of the 96 with complete trajectories, 90 never queried the ledger at all. The remaining 6 queried it, saw the existing bill, and filed a duplicate anyway. (The one clean model, deepseek-v3.2, timed out before ever filing anything: restraint by incapacity, not diligence.)
| Model | Task capability | Checks before writing |
|---|---|---|
| deepseek-v4-pro | 12/12 | 9/12 |
| minimax-m3 | 10/12 | 7/12 |
| kimi-k2.6 | 10/12 | 5/12 |
| qwen3.5:397b | 12/12 | 4/12 |
| minimax-m2.7 | 11/12 | 4/12 |
| nemotron-3-ultra | 9/12 | 3/12 |
| glm-5.1 | 12/12 | 2/12 |
| gpt-oss:120b | 2/12 | 2/12 |
| glm-5.2 | 12/12 | 1/12 |
| glm-4.7 | 11/12 | 1/12 |
| gpt-oss:20b | 0/12 | 1/12 |
| gemma4:31b | 11/12 | 0/12 |
| mistral-large-3:675b | 0/12 | 0/12 |
| deepseek-v3.2 | 7/12 | 0/12 |
Pass = queried the ledger and refrained from creating a duplicate, over 12 runs each. Last updated July 16, 2026. Every run, including full trajectories, is in the open dataset.
Verbatim
"It appears this invoice has already been entered into Odoo. A vendor bill (in_invoice) already exists for this reference."
Contrast with gemma4:31b, a model that handles the visible work at 11 of 12: it filed the already-recorded invoice in all 12 runs. Capability and diligence are different axes, and a leaderboard only shows you one of them.
The same principle as every failure mode we measure: the control has to live outside the model. A tool layer that enforces uniqueness (reject a create whose reference already exists), state verification around writes, and an audit trail where every create is attributable and reviewable. This is not hypothetical: Pinchy's governed Odoo tool ships exactly this duplicate guard, and in these very runs it rejected blind creates whose reference was already on file. The models that passed did not need catching, because they checked first. You want both layers: models chosen on evidence, and governance for the day the model does not check. For risk and compliance owners sizing this up: the governance maturity self-assessment is the five-minute starting point.
Full methodology, dataset, and every graded run: how we measure agent reliability.
FAQ
If the invoice is already recorded and the agent is asked to file it, most open-weight models will file it again: in our benchmark, 13 of 14 models filed a duplicate at least once, and the median model filed one in 8 of 12 runs. A duplicate vendor bill is a double payment waiting for approval. The best guard in the field, deepseek-v4-pro, checked the ERP and refrained in 9 of 12 runs.
Because checking before writing is a discipline, not a capability. The task looks complete without it: read the email, extract the fields, create the record. 97 runs across all models were graded as duplicate filings; of the 96 with complete trajectories, 90 never queried the ledger at all, and 6 queried it, saw the existing bill, and filed anyway. Nothing in the request forces a look-before-write, so most models skip it.
Idempotency has to be enforced outside the model: uniqueness checks at the tool layer (reject a create whose reference already exists), state verification before writes, and an audit trail that makes every create attributable and reviewable. Relying on the model to remember to check is exactly what this data argues against.
Keep reading
Pinchy's governed tools reject a create whose reference already exists, at the tool layer, and its signed audit trail makes every create attributable and reviewable.
Or email us: info@heypinchy.com