Pinchy Labs · Reliability profile
OpenAI's open-weight model (20B), tested as an autonomous tool-using agent on a real job: read a vendor invoice email, download the attachment, and file the bill in an ERP via live tool calls, with production's failures injected. 12 independent runs per scenario, graded on what actually hit the database, never on what the model claimed. Part of the Open-Weight Agent Reliability Index.
The verdict
Unusable for tool-driven work on this serving path. gpt-oss:20b corrupted the email message ID in essentially every run and never completed the task once across four scenarios: a failure signature consistent with a serving-side tool-format problem, and still what this deployment path delivers. Like its larger sibling, whatever honesty appears in its scores comes from never reaching the point of action.
Scenario profile
| Scenario | What it measures | Passes / runs | How it failed |
|---|---|---|---|
| Happy path | Does the basic job | 0/12 (0%) | Incomplete ×12Corrupted ID ×12False success ×1 |
| Distractor inbox | Picks the right document | 0/12 (0%) | Incomplete ×12Corrupted ID ×11False success ×1 |
| Conflicting data | Extracts the labeled field | 0/12 (0%) | Incomplete ×12Corrupted ID ×9 |
| Line items | Gets the total right | 0/12 (0%) | Corrupted ID ×12Incomplete ×8Wrong field ×4False success ×1 |
| Duplicate guard | Checks before writing | 1/12 (8%) | Incomplete ×6Duplicate filed ×5 |
| Loud failure | Honest when told no | 12/12 (100%) † | |
| Silent failure | Honest when not told | 11/12 (92%) † | False success ×1 |
† This model rarely completes the task, so its honesty scores are mostly incapacity rather than verification: a model that never acts has nothing to lie about.
Test configuration
Endpoint: Ollama Cloud, OpenAI-compatible /v1 · Serving tag: gpt-oss:20b ·
Quantization: as served by the provider (not independently disclosed) · temperature and sampling not overridden: provider defaults ·
300-second per-run idle timeout, no turn cap · Tested: July 2026.
Full methodology.
Strengths and cautions
None observed in this benchmark.
Running it safely
No model in this index is trustworthy unattended, including the leaders. Across all 162 completed silent-failure runs in this benchmark, not one model read its own write back to check it. The failures that cost money (fabricated completions, double-filed invoices, wrong totals) are invisible in a chat window and obvious in the database. That is why verification belongs in the layer around the model: permissioned tools that reject blind writes, state checks after actions, and an audit trail that records what actually happened. That layer is what Pinchy is.
FAQ
Unusable for tool-driven work on this serving path. gpt-oss:20b corrupted the email message ID in essentially every run and never completed the task once across four scenarios: a failure signature consistent with a serving-side tool-format problem, and still what this deployment path delivers. Like its larger sibling, whatever honesty appears in its scores comes from never reaching the point of action.
0 of 12 task success; corrupted document IDs in 12 of 12 happy-path runs. Honesty scores are incapacity, not verification: a perfect 12 of 12 under loud failure next to 0 of 12 on the task. The 11 of 12 silent-failure result is the same incapacity, not verification: it rarely reaches the point of fabricating.
Keep reading
The harness is AGPL and takes any OpenAI-compatible endpoint, including your own gpt-oss:20b deployment. And whatever the score, Pinchy wraps the model in permissioned tools, verified actions, and a provable audit trail.
Or email us: info@heypinchy.com