Pinchy Labs · Scenario 3 of 7

Conflicting data: Does it extract the labeled field, or the loudest one?

One invoice, two traps. The subject line and a reference line shout a wrong number; the real invoice number sits once in the body, behind the label "Invoice number:". An order date and a due date flank the actual invoice date. Skimming files the wrong record. Part of the Open-Weight Agent Reliability Index: 12 independent runs per model, graded on what actually hit the database, never on what the model claimed.

What this scenario found

Labels beat prominence for most capable models, but the exceptions are consistent enough to name: glm-5.1 grabbed the loud wrong number in a third of its runs.

Last updated July 16, 2026 · 14 models measured · methodology · machine-readable results · raw trajectories on GitHub

The setup

What the agent sees.

The instruction, verbatim:

"Read the latest invoice email from Hetzner and enter it into Odoo as a vendor bill."

What counts as a pass.

Why it matters: Invoices are full of numbers that look like the number: order references, customer IDs, due dates. A model that keys on prominence instead of labels corrupts exactly the fields your accounting reconciles on.

The results

7 of 14 models strong, 6 unreliable.

Passes over 12 independent runs per model, sorted by pass rate. At 12 runs per cell, 11/12 versus 12/12 is within noise: read bands, not points.

  1. kimi-k2.6 Moonshot AI
    12/12
  2. gemma4:31b Google · 31B
    12/12
  3. qwen3.5:397b Alibaba · 397B
    12/12
  4. glm-4.7 Zhipu AI
    12/12
  5. deepseek-v4-pro DeepSeek
    11/12
  6. glm-5.2 Zhipu AI
    11/12
  7. minimax-m2.7 MiniMax
    11/12
  8. glm-5.1 Zhipu AI
    8/12
  9. minimax-m3 MiniMax
    5/12
  10. gpt-oss:120b OpenAI · 120B
    5/12
  11. nemotron-3-ultra NVIDIA
    1/12
  12. deepseek-v3.2 DeepSeek
    1/12
  13. mistral-large-3:675b Mistral AI · 675B
    0/12
  14. gpt-oss:20b OpenAI · 20B
    0/12
11–12 of 12 strong 8–10 of 12 slips 7 or fewer unreliable click a model for its full profile

How the field failed, across all graded runs

Incomplete ×32Timeout ×18Corrupted ID ×14Wrong field ×13Loop ×2

Test configuration

Endpoint: Ollama Cloud, OpenAI-compatible /v1 · Quantization: as served by the provider (not independently disclosed) · temperature and sampling not overridden: provider defaults · 300-second per-run idle timeout, no turn cap · Tested: July 2026. Full methodology.

Reading the column

What we saw.

Most capable models pass cleanly: gemma4:31b, kimi-k2.6, qwen3.5, and glm-4.7 were perfect over 12 runs. The interesting failure is glm-5.1, which took the prominent reference number in 4 of 12 runs, more than any other capable model, while its sibling glm-5.2 slipped only once. Below the pack, minimax-m3 mixed wrong fields with timeouts, and nemotron-3-ultra fell to 1 of 12.

Keep reading

Related

The full index

All 14 models across all 7 scenarios in one matrix.

Methodology

State-based grading, the failure taxonomy, and how to reproduce every number.

Next: Line items

Same invoice, harder standard: the bill must be entered with its line items so the ERP computes the same EUR 47.60 the invoice states.

The failures this scenario measures are the ones governance catches.

Pinchy wraps any of these models in permissioned tools that reject blind writes, state checks after actions, and a signed audit trail that makes every claimed action checkable against a logged one.

Benchmark your own endpoint Book a demo

Or email us: info@heypinchy.com