Pinchy Labs · Scenario 4 of 7

Line items: Does the total actually add up?

Same invoice, harder standard: the bill must be entered with its line items so the ERP computes the same EUR 47.60 the invoice states. Header-only entries, net/gross confusion, and invented line items all fail. This is where data entry becomes accounting. Part of the Open-Weight Agent Reliability Index: 12 independent runs per model, graded on what actually hit the database, never on what the model claimed.

What this scenario found

The widest spread in the index: from flawless (deepseek-v4-pro) to zero correct totals in 12 attempts (minimax-m3). If your agents write structured records, this column reorders the shortlist.

Last updated July 16, 2026 · 14 models measured · methodology · machine-readable results · raw trajectories on GitHub

The setup

What the agent sees.

The instruction, verbatim:

"Read the latest invoice email from Hetzner and enter it into Odoo as a vendor bill. Record the invoice line item(s) so the bill's total matches the invoice amount."

What counts as a pass.

Why it matters: A bill whose total does not match the invoice is not a smaller success; it is a wrong liability in the books. Net/gross confusion is the classic slip: both numbers are on the document, and the model picks the wrong one.

The results

2 of 14 models strong, 9 unreliable.

Passes over 12 independent runs per model, sorted by pass rate. At 12 runs per cell, 11/12 versus 12/12 is within noise: read bands, not points.

  1. deepseek-v4-pro DeepSeek
    12/12
  2. gemma4:31b Google · 31B
    11/12
  3. kimi-k2.6 Moonshot AI
    10/12
  4. glm-5.2 Zhipu AI
    9/12
  5. glm-5.1 Zhipu AI
    9/12
  6. glm-4.7 Zhipu AI
    7/12
  7. qwen3.5:397b Alibaba · 397B
    6/12
  8. minimax-m2.7 MiniMax
    5/12
  9. gpt-oss:120b OpenAI · 120B
    3/12
  10. nemotron-3-ultra NVIDIA
    1/12
  11. mistral-large-3:675b Mistral AI · 675B
    1/12
  12. minimax-m3 MiniMax
    0/12
  13. gpt-oss:20b OpenAI · 20B
    0/12
  14. deepseek-v3.2 DeepSeek
    0/12
11–12 of 12 strong 8–10 of 12 slips 7 or fewer unreliable click a model for its full profile

How the field failed, across all graded runs

Wrong field ×49Incomplete ×35Corrupted ID ×21Timeout ×5Loop ×5False success ×1

Test configuration

Endpoint: Ollama Cloud, OpenAI-compatible /v1 · Quantization: as served by the provider (not independently disclosed) · temperature and sampling not overridden: provider defaults · 300-second per-run idle timeout, no turn cap · Tested: July 2026. Full methodology.

Reading the column

What we saw.

The widest spread in the index. deepseek-v4-pro is perfect and gemma4:31b nearly so, while qwen3.5, flawless on every other capability column, drops to 6 of 12 on net/gross confusion. minimax-m3 never produced a single correct total in 12 runs. Capability columns usually move together; this one does not.

Keep reading

Related

The full index

All 14 models across all 7 scenarios in one matrix.

Methodology

State-based grading, the failure taxonomy, and how to reproduce every number.

Next: Duplicate guard

The invoice is already in the ERP, filed and posted, before the agent starts.

The failures this scenario measures are the ones governance catches.

Pinchy wraps any of these models in permissioned tools that reject blind writes, state checks after actions, and a signed audit trail that makes every claimed action checkable against a logged one.

Benchmark your own endpoint Book a demo

Or email us: info@heypinchy.com