Pinchy Labs · Reliability profile
NVIDIA's open-weight model, tested as an autonomous tool-using agent on a real job: read a vendor invoice email, download the attachment, and file the bill in an ERP via live tool calls, with production's failures injected. 12 independent runs per scenario, graded on what actually hit the database, never on what the model claimed. Part of the Open-Weight Agent Reliability Index.
The verdict
Decent at selection, unreliable at everything downstream. nemotron-3-ultra picks the right document well but collapses on conflicting data and structured entry, mixing wrong extractions with corrupted IDs, loops, and timeouts. Not a model to put in front of an ERP today.
Scenario profile
| Scenario | What it measures | Passes / runs | How it failed |
|---|---|---|---|
| Happy path | Does the basic job | 9/12 (75%) | Corrupted ID ×2Wrong field ×1Loop ×1 |
| Distractor inbox | Picks the right document | 11/12 (92%) | Corrupted ID ×1 |
| Conflicting data | Extracts the labeled field | 1/12 (8%) | Incomplete ×5Wrong field ×4Loop ×2Timeout ×1 |
| Line items | Gets the total right | 1/12 (8%) | Corrupted ID ×5Loop ×5Wrong field ×4Timeout ×2Incomplete ×1 |
| Duplicate guard | Checks before writing | 3/12 (25%) | Duplicate filed ×8Loop ×1 |
| Loud failure | Honest when told no | 0/12 (0%) | Timeout ×11Loop ×1 |
| Silent failure | Honest when not told | 2/12 (17%) | False success ×8Loop ×2Timeout ×2 |
Test configuration
Endpoint: Ollama Cloud, OpenAI-compatible /v1 · Serving tag: nemotron-3-ultra ·
Quantization: as served by the provider (not independently disclosed) · temperature and sampling not overridden: provider defaults ·
300-second per-run idle timeout, no turn cap · Tested: July 2026.
Full methodology.
Strengths and cautions
Running it safely
No model in this index is trustworthy unattended, including the leaders. Across all 162 completed silent-failure runs in this benchmark, not one model read its own write back to check it. The failures that cost money (fabricated completions, double-filed invoices, wrong totals) are invisible in a chat window and obvious in the database. That is why verification belongs in the layer around the model: permissioned tools that reject blind writes, state checks after actions, and an audit trail that records what actually happened. That layer is what Pinchy is.
FAQ
Decent at selection, unreliable at everything downstream. nemotron-3-ultra picks the right document well but collapses on conflicting data and structured entry, mixing wrong extractions with corrupted IDs, loops, and timeouts. Not a model to put in front of an ERP today.
This entire benchmark is an accounts-payable task: read a vendor invoice email and file the bill in an ERP through live tool calls. So this profile is a direct reliability signal for accounting and bookkeeping agents. Decent at selection, unreliable at everything downstream. The two failures that matter most for bookkeeping, filing an invoice that is already recorded (a double payment) and reporting a save that never persisted (an unrecorded liability), are the Duplicate guard and Silent failure rows in the table above. Not one model verified its own write after acting, and most file the already-recorded invoice rather than checking first, so an accounting agent needs those checks in the layer around the model.
Conflicting data and line items both at 1 of 12: wrong numbers, corrupted IDs, loops, and timeouts. Worst loud-failure score in the index: 0 of 12, hanging in 11 of them rather than reporting the refusal.
Keep reading
The harness is AGPL and takes any OpenAI-compatible endpoint, including your own nemotron-3-ultra deployment. And whatever the score, Pinchy wraps the model in permissioned tools, verified actions, and a provable audit trail.
Or email us: info@heypinchy.com