Four models did best, and their scores are too close to call. Their prices are not. Running every test once cost $1.34 on DeepSeek V4.1 Flash and $19.83 on GPT-6 Astra.
The short version
Among the four best models, pick on price. The cheapest does the same work for a fifteenth of the cost.
Some models guess when they should say “I can’t tell,” and say they are sure. Avoid them where a wrong answer is expensive.
Two database mistakes trip up every model. Your instructions have to warn the agent about them.
Check that a model actually uses its tools. Two of the fourteen often don’t.
The top four score about the same. The most expensive costs fifteen times the cheapest.
Score
Cost to run every test once
Score is the average pass rate over the 12 kinds of work, with larger kinds counting more. Cost is what running all 310 tasks once cost at each provider’s list price.
Four models share first place
We only call one model better than another if the gap is too big to be luck. By that test, no model beats Gemini 3.8 Flash, GPT-6 Astra, DeepSeek V4.1 Flash or GPT-5.6-sol. Their order among themselves could flip on a different set of tasks.
Below them the models spread out. Nemotron 3.5 Lightning scored lowest, far behind the rest.
Score for all 14 models
Blue: the four models that no other model beats by more than luck would allow.
What to doPick among the top four on price and on the work you need. Don’t pay for the name at the top of the list.
Some models guess, and say they’re sure
In 15 of the tasks, a company name matched several records. The right answer was “this is ambiguous.” Some models said so every time. Others picked one record and claimed it was the one.
Every answer also came with the model’s own estimate of how likely it was to be right. The models that guessed most still said they were almost certain.
The models that guessed said they were sure
On 15 tasks where the right answer was “this is ambiguous”
How often it was rightHow sure it said it was
An orange line marks a model that was far more sure than it was right. Nemotron 3 Super was never right on these tasks, and said it was 95% sure.
What to doIf a confident wrong match is expensive, use GPT-6 Astra, Gemini 3.8 Flash, or GLM 5.3 Flash through a host that lets it reason.
Two database mistakes every model makes
Two kinds of database question beat almost every model. Neither takes special knowledge. Both take one extra check that no model makes on its own.
One company, two names. A drug company’s trials are filed under Seagen, a wholly owned subsidiary of Pfizer in some rows and plain Seagen Inc. in others. Asked for all of Pfizer’s trials, models found the first kind and missed the second. The best model got 39% right.
A sign in its own column. A p-value is stored as 0.05 with its < or > in a separate column. Models read the number and ignored the sign. The best model got 62% right.
What to doNo choice of model fixes these. If your data has either problem, say so in the agent’s instructions.
Check that a model uses its tools
Most of these tasks need a tool: a database, a company register, a search. Two models fall short. Nemotron 3.5 Lightning often answers from memory without running the query. Gemini 3.5 Flash-Lite starts multi-step jobs and doesn’t finish them.
Two models finish far fewer multi-step tool jobs
Share of 17 jobs completed, each spanning several tools
What to doBefore trusting a model’s score, test it on work that needs its tools.
One good sign: across all 6,066 answers, no model ever claimed to use a tool it hadn’t used.
More detail
Every model on every kind of work
Gemini 3.8 Flash
GPT-6 Astra
DeepSeek V4.1 Flash
GPT-5.6-sol
Qwen3.8 Flash
GLM 5.3 Flash (hosted)
DeepSeek V4 Flash
Qwen3.8 27B
GLM 5.3 Flash
GPT-5.6-luna
MiniMax M3
Nemotron 3 Super
Gemini 3.5 Flash-Lite
Nemotron 3.5 Lightning
Analytic judgment
93
93
100
93
87
73
83
67
80
77
67
87
80
0
Constrained deduction
60
46
65
51
54
52
53
58
46
40
47
36
41
34
Corporate hierarchy
100
100
100
100
100
100
100
100
100
100
100
92
100
100
Entity resolution — ambiguous evidence
100
100
80
87
87
100
33
60
63
29
47
0
67
20
Entity resolution — distinct entities
100
100
100
100
100
93
100
100
94
100
100
100
93
79
Entity resolution — same entity
89
78
100
89
100
78
94
100
87
89
89
100
89
56
Filing research
100
100
100
100
100
100
100
100
75
100
100
75
50
0
Learned-prompt distillation
100
97
97
97
97
94
95
94
95
82
85
94
61
56
Multi-step tool chains
100
100
82
100
100
76
83
100
92
91
82
88
53
29
Register lookup
100
83
92
92
100
100
94
92
88
98
100
100
100
58
Relational analytics
52
62
54
52
47
47
54
42
50
50
43
43
40
12
URL extraction
100
100
81
100
76
76
86
86
71
80
81
86
48
52
Pass rate in percent, models in the order they scored. Darker is higher.
Which model for which job
DeepSeek V4.1 Flash, for mixed agent work: one of the top four, and the cheapest of them. Not the one for multi-step tool jobs, where five models finish every one.
Qwen3.8 Flash, for the same work for less: fifth overall, for $0.85 against DeepSeek’s $1.34. It finished every multi-step tool job.
GLM 5.3 Flash, for ambiguous inputs: through a host that lets it reason, it said “ambiguous” every time it should have.
GPT-5.6-luna, for cheap work on evidence you hand it: $0.62 for every test, and it did well on puzzles when given every clue. It did badly when it had to search for them.
The same model through two providers
GLM 5.3 Flash is an open model, so we ran it through two hosts. Overall the two scored within three points. On the ambiguous names they were 37 points apart.
Both found the same records. Only one said the name was ambiguous. That host let the model write more than twice as much reasoning per task. A score for an open model is a score for one way of running it.
How we tested, and the limits
Every answer was marked right or wrong against a fixed answer key. There was no partial credit and no AI grading. Every model ran the same agents at its provider’s default settings. We set no reasoning effort or thinking budget.
We picked tasks where models disagree, plus tasks every model fails. So these scores are lower than on other benchmarks and can’t be compared with them. They can put these 14 models in order and size the gaps between them.
Cost is each model’s list price on 2026-09-13, applied to the tokens it used. It counts only the answer we graded, so a real bill is at least this much. Qwen3.8 27B runs on our own hardware and has no list price.