Pinchy Labs · Scenario 2 of 7

Distractor inbox: Does it pick the right document?

Now there are two Hetzner invoices, same sender, both with a PDF: the EUR 47.60 cloud invoice the request refers to, and a EUR 89.00 dedicated-server invoice sitting next to it. Filing the wrong one is not a crash; it is wrong data with a paper trail. Part of the Open-Weight Agent Reliability Index: 12 independent runs per model, graded on what actually hit the database, never on what the model claimed.

What this scenario found

Document selection is largely solved for capable models. The failures in this column are capability collapse (timeouts, corrupted handles), not wrong picks.

Last updated July 16, 2026 · 14 models measured · methodology · machine-readable results · raw trajectories on GitHub

The setup

What the agent sees.

The instruction, verbatim:

"There are a couple of Hetzner invoices in the inbox. Enter the one for our Hetzner Cloud services into Odoo as a vendor bill."

What counts as a pass.

Why it matters: Real inboxes never contain exactly one document. Selection errors are quiet: the workflow "succeeds", the ERP fills with plausible wrong records, and nobody notices until reconciliation.

The results

8 of 14 models strong, 4 unreliable.

Passes over 12 independent runs per model, sorted by pass rate. At 12 runs per cell, 11/12 versus 12/12 is within noise: read bands, not points.

  1. deepseek-v4-pro DeepSeek
    12/12
  2. kimi-k2.6 Moonshot AI
    12/12
  3. glm-5.2 Zhipu AI
    12/12
  4. glm-5.1 Zhipu AI
    12/12
  5. qwen3.5:397b Alibaba · 397B
    11/12
  6. glm-4.7 Zhipu AI
    11/12
  7. minimax-m2.7 MiniMax
    11/12
  8. nemotron-3-ultra NVIDIA
    11/12
  9. gemma4:31b Google · 31B
    10/12
  10. minimax-m3 MiniMax
    10/12
  11. gpt-oss:120b OpenAI · 120B
    4/12
  12. deepseek-v3.2 DeepSeek
    1/12
  13. mistral-large-3:675b Mistral AI · 675B
    0/12
  14. gpt-oss:20b OpenAI · 20B
    0/12
11–12 of 12 strong 8–10 of 12 slips 7 or fewer unreliable click a model for its full profile

How the field failed, across all graded runs

Incomplete ×29Corrupted ID ×17Timeout ×11Wrong field ×5False success ×2

Test configuration

Endpoint: Ollama Cloud, OpenAI-compatible /v1 · Quantization: as served by the provider (not independently disclosed) · temperature and sampling not overridden: provider defaults · 300-second per-run idle timeout, no turn cap · Tested: July 2026. Full methodology.

Reading the column

What we saw.

The strongest column in the matrix. kimi-k2.6, deepseek-v4-pro, and both recent GLMs picked and filed the right invoice in all 12 runs each, and most of the field is close behind. The failures at the bottom are not selection errors: deepseek-v3.2 timed out in 11 of 12 runs, and the gpt-oss pair never got as far as choosing.

Keep reading

Related

The full index

All 14 models across all 7 scenarios in one matrix.

Methodology

State-based grading, the failure taxonomy, and how to reproduce every number.

Next: Conflicting data

One invoice, two traps.

The failures this scenario measures are the ones governance catches.

Pinchy wraps any of these models in permissioned tools that reject blind writes, state checks after actions, and a signed audit trail that makes every claimed action checkable against a logged one.

Benchmark your own endpoint Book a demo

Or email us: info@heypinchy.com