Does MMLU matter when choosing an LLM for production?

Barely, for most production agents. MMLU measures multiple-choice knowledge and reasoning in isolation, with no tools, no external data and no structured output. A production agent typically answers by calling tools against live data, so what determines success is tool-calling accuracy, instruction-following, structured-output reliability and grounding discipline. MMLU is a useful floor check — a very low score is a red flag — but a high one predicts very little about your workload.

Why can't I compare tau-bench scores between vendors directly?

Because the conditions differ and the footnotes carry them. Some published scores are measured with extended thinking enabled, or with an added prompt addendum, or with tools available where another figure was measured without. And tau-bench and tau-2-bench are different benchmarks — a tau-2-bench retail score is not comparable cell-to-cell with a tau-bench retail score, even though both are labelled 'retail'.

How do I evaluate an LLM for my own production use case?

Run the candidates through your real endpoint with your real tools against real data, computing ground truth from your database so every asserted figure can be verified. Include adversarial traps for nonexistent entities and injected instructions, mirror every scenario into each language you support, repeat each scenario several times, and report confidence intervals rather than single scores. Then weigh cost and latency, because those decide whether the feature is usable.

Can a model with lower benchmark scores be the better production choice?

Yes, and it is common. In our own selection, the candidate with the stronger general and agentic public scores lost on the workload that mattered: it failed more grounding and safety traps, consumed far more input tokens, cost roughly ten times more per answered query and delivered tokens several times slower. General capability is one input among several, and it is rarely the one that decides whether a feature ships.

Public LLM Benchmarks Won't Tell You Which Model to Ship

Q: Which LLM benchmarks are relevant for agents and tool use?

tau-bench and BFCL are the closest public proxies. tau-bench evaluates multi-turn task completion in simulated domains such as retail and airline, where the model must use tools and follow policy rules. BFCL (the Berkeley Function-Calling Leaderboard) evaluates function-calling accuracy directly, including multiple and parallel calls. Both are far more predictive than MMLU for agentic products, but neither uses your tools, your schema or your data.

July 24, 20267 min readBy Rafa Jiménez

Leaderboard scores measure a model alone in a room with a multiple-choice exam. Your product needs it to call the right tool, obey a schema, refuse to invent a number, and come back fast enough that someone waits for the answer. Those are different skills, and the public numbers barely test them.

Key Takeaways

MMLU and GPQA measure knowledge and reasoning in isolation. Production agents live or die on tool-calling, instruction-following, structured output and grounding.
tau-bench and BFCL are the relevant agentic benchmarks — but read the footnotes: scores are measured with or without tools, with or without extended thinking, and tau-bench and tau-2-bench are different benchmarks.
The model with the best general scores can still be the wrong pick once grounding discipline, effective delivery speed and cost are weighed together.
The only benchmark that settles it is your own: real endpoint, real tools, real data, ground truth from your database, adversarial traps, repeated runs.

Every model launch arrives with a table. MMLU, GPQA Diamond, SWE-bench, AIME, a tau-bench column if the model is being sold as agentic. The numbers are real, and reading them is a genuine skill. But the table answers a question most teams are not asking.

We went through this while choosing the model behind Seal AI, the natural-language analytics assistant inside SealMetrics. The public numbers narrowed the field. They did not pick the winner, and on the decisive dimension they pointed the wrong way.

What the famous benchmarks actually measure

MMLU and MMLU-Pro are multiple-choice exams across academic and professional subjects. GPQA Diamond is a set of graduate-level science questions designed to be resistant to web search. AIME is competition mathematics. SWE-bench Verified asks a model to resolve real GitHub issues.

Each is a legitimate measure of something. None of them involves the model being handed a catalogue of tools, deciding which to call, and narrating the result without embellishing it. That is the actual job description for most production agents.

Benchmark	What it measures	What your product needs instead
MMLU / MMLU-Pro	Multiple-choice knowledge and reasoning, no tools, no external data	Correct tool selection from a large inventory, turn after turn
GPQA Diamond	Graduate-level science reasoning, search-resistant	Reading a returned table correctly and not rounding a number into fiction
AIME	Competition mathematics	Simple arithmetic on retrieved figures, stated with the right period and timezone
SWE-bench Verified	Resolving real software issues in a repository	Emitting output that validates against your response schema every time
SimpleQA	Open-world factual recall from parameters, and how often the model invents	Grounding: refusing to answer from memory when the data is in the prompt
tau-bench / BFCL	Multi-turn task completion and function-calling accuracy in simulated domains	The same skills — measured on your tools, schema and data

That SimpleQA row deserves a moment, because it shows why a bad score can be irrelevant. The model we shipped, gpt-oss-120b, scores 0.168 accuracy on SimpleQA with a 0.782 hallucination rate. Read cold, that looks disqualifying. Read in context, it describes weak open-world factual recall — how much trivia the weights happen to contain. Our assistant never answers from its weights. The data travels in the prompt, retrieved by tools, and every figure is checked against the database. The benchmark measures a capability the product does not use.

The benchmarks that are relevant — and their footnotes

Two public families do track what agentic products need. tau-bench puts a model in a multi-turn simulated domain (retail, airline) where it must use tools and respect policy rules to complete a task. BFCL, the Berkeley Function-Calling Leaderboard, measures function-calling accuracy directly, including multiple and parallel calls.

Use them. But read them like a lawyer, because vendor tables put the conditions in the footnotes and the numbers in the cells, and people copy cells.

With tools or without. gpt-oss-120b scores 80.1 on GPQA Diamond without tools and 80.9 with tools. Its AIME 2025 figure of 97.9 is a with-tools number. Quoting one against a rival's other is not a comparison, it is a category error.
With extended thinking, or with a prompt addendum. Claude Sonnet 4's tau-bench retail score of 80.5 is measured with extended thinking enabled plus a prompt addendum, per Anthropic's own footnote. That is a legitimate published configuration, clearly disclosed — and it is not the same configuration as a plain single-pass run.
Different benchmark, same word. Claude Sonnet 4.5's 86.2 retail figure is tau-2-bench, not tau-bench. Both say "retail". They are not the same test, and lining them up in one column produces a ranking that does not exist.

For scale, some reference points that are comparable to each other: GPT-4o sits around 60.4 to 61.2 on tau-bench retail; gpt-oss-120b at 67.8; qwen3-235b-a22b-2507 at 71.3. Those three you can line up. The Claude figures above you cannot, not without restating the conditions.

We re-verified every public figure in our own model-selection write-up against primary sources — vendor model cards and benchmark publications rather than secondary round-ups. Two of them turned out to be wrong in our first draft, both from exactly the failure modes listed above. If it happened to us while we were being careful, it is happening in the comparison table you are reading somewhere else.

Speed is a benchmark too, and it is not the one you think

Cross-provider medians published by Artificial Analysis (retrieved 23 July 2026 — these drift daily) put gpt-oss-120b at 262.8 output tokens per second with a 0.89 second time to first token, Gemini 2.5 Flash at 201.9, GPT-4.1 at 114.7, Qwen3-235B at 56.7 with a 2.32 second time to first token, and Claude Sonnet 4.5 at 42.0. GPT-5 at high reasoning effort has a time to first token above 100 seconds — a fine trade for a batch job, unusable for interactive chat.

Two warnings about numbers like these. First, they are medians across providers; what you get depends on the host you actually use. On our own infrastructure gpt-oss-120b delivers an effective 75 to 90 output tokens per second inside real tool loops, which is the number that governs the user experience — not the headline median. Second, an agentic answer is several model calls plus tool round-trips, so time-to-first-token matters more than peak throughput for whether the interface feels alive.

A concrete case where the leaderboard pointed the wrong way

On public numbers, qwen3-235b-a22b-2507 beats gpt-oss-120b on almost every axis we care about: 93.1 on MMLU-Redux against 90.0 on MMLU, 83.0 against 80.8 on MMLU-Pro, 71.3 against 67.8 on tau-bench retail, plus 70.9 on BFCL v3 and the strongest multilingual score in the open set. If a table decided this, the table decided for qwen.

Then we ran both through the real product: 18 scenarios in Spanish and English, three passes, three models, 162 live queries against a real account's data with the full 63-tool inventory and ground truth computed from the database.

gpt-oss-120b passed 18 of 18 grounding and injection traps and verified 144 of 144 asserted facts. qwen3-235b passed 15 of 18 — and its failures included reproducing an injected instruction in two of three English attempts while ignoring the identical attack in all nine Spanish runs. It also consumed 2.2 times the input tokens, worked out roughly ten times more expensive per answered query, and delivered 21 to 23 effective output tokens per second against our winner's 75 to 90.

None of that is visible in a leaderboard. It is not a criticism of the leaderboard either — those benchmarks were never designed to predict cost per answered query on a 63-tool analytics workload in two languages. They answered their question correctly. It was simply a different question from ours.

Five steps to evaluate a model on your own workload

Use public benchmarks as a filter, not a decision. They are excellent at eliminating candidates that are clearly not capable enough. Stop using them the moment two candidates are both plausible.
Test through your real endpoint. Same product code, same tool inventory, same data; switch only the model by configuration. A sandbox with three toy tools tells you nothing about behaviour with sixty.
Compute ground truth from your database. Pull the true figures at the start of the run so every number the model asserts is checked automatically. Without ground truth you are grading vibes.
Add adversarial scenarios and repeat them. Ask about entities that do not exist. Ask about periods with no data. Plant an injected instruction with a canary. Mirror all of it into every language you support, run each scenario several times, and report confidence intervals rather than single scores.
Weigh cost and effective speed last, and seriously. Two models that are both correct enough are separated by tokens consumed and time to answer. That is where an order-of-magnitude difference usually hides.

And publish your defects. Our run contained a badly worded question — "traffic by device for the last month" admits two readings, and every single fact miss in the entire run traces back to it. That is our bug, not a model hallucination, and saying so is what makes the rest of the numbers worth reading.

The bottom line

Public benchmarks tell you whether a model is capable. They do not tell you whether it is suitable. Suitability is a function of your tools, your schema, your languages, your latency budget and your tolerance for a confidently wrong number — and none of those appear in a launch table.

The longer version of this argument, with the full model-selection reasoning, is in The Best LLM for Data Analytics Isn't the One With the Highest Benchmark. The market audit behind it, including the EU-sovereign hosting constraint that shaped the shortlist, is documented in our model selection write-up.