How do I choose an LLM for my product?

Choose it on your own workload, not on a leaderboard. Run the candidates through your real stack — your actual endpoint, your actual tool inventory, your actual data — and score them on the behaviours your product depends on: correct tool calls, grounding in the data returned, clean structured output, and refusal behaviour. Ship the winner behind a configuration switch so replacing it later costs a deploy, not a rewrite.

Gemma vs Mistral vs gpt-oss for tool calling — which is better?

In our own testing against a 63-tool analytics inventory, gpt-oss-120b had the best tool-calling of the three. A small mixture-of-experts Gemma variant with roughly 4B active parameters per token was too weak for the job and degenerated into repetition loops. Mistral Small 3.2 (dense 24B) was the fastest and stopped the loops, but produced terse answers, occasional groundless refusals and broken formatting in real use.

Why do small mixture-of-experts models fail at function calling?

Because only a fraction of the parameters are active per token. A model advertised at 26B total but activating around 4B per token has roughly 4B worth of reasoning available at each step. Planning a multi-step tool call across dozens of functions, then synthesising the results, is exactly the kind of sustained work that thin active capacity fails at — in our case by collapsing into repetition loops mid-answer.

Is it bad if a SaaS vendor changes the AI model behind its product?

Not by itself — what matters is why and how. A change driven by measurement on the vendor's own workload, with the evidence published, is a sign the vendor is testing. A change driven by whichever model was announced most recently is not. Ask what test the new model passed that the old one failed, and whether the old results are still available.

What is gpt-oss-120b and why use it for analytics?

gpt-oss-120b is an open-weight model under the Apache 2.0 licence: 117B total parameters with 5.1B active per token, a 128k context window, native MXFP4 quantization that fits a single 80GB GPU, and first-class vLLM support. For grounded analytics it wins on tool-calling reliability rather than general knowledge, because the data travels in the prompt and the model only has to operate on it.

We Changed Our AI Model Twice in Three Weeks — And That's the Point

July 24, 20266 min readBy Rafa Jiménez

In three weeks, the model behind Seal AI changed twice. Our first pick was wrong, the second was better and still not good enough, and the third is running today. Every swap came from evidence in our own product — not from a launch announcement.

Key Takeaways

Our first pick was wrong. A mixture-of-experts model with only about 4B active parameters per token could not plan tool calls across a 63-function inventory and degenerated into repetition loops.
The second pick stopped the loops and was the fastest of the lineup, but real use exposed terse answers, groundless refusals and broken formatting.
The third — gpt-oss-120b — had the best tool-calling of the three, zero refusals and zero format failures, at quality on par with a model costing roughly four times more.
Two swaps in three weeks is not instability. It is what measuring looks like: each change was triggered by a failure we observed, then re-verified by a 162-query bilingual benchmark.

Vendors do not usually publish this part. The model behind an AI feature gets announced once, in the launch post, and any later change happens quietly. We are doing the opposite, because the changes are the most useful thing we can tell you about how the feature was built.

Seal AI is the private AI layer inside SealMetrics: you ask a question about your analytics in plain language, and it answers by calling tools against your data — 63 of them, from overviews and channels to funnels and segments. That job punishes models in ways general benchmarks never show. Here is what happened when we tried three of them.

The timeline

2 July 2026 — gemma-4-26b-a4b. First pick. Replaced after it proved too weak for function calling and degenerated into repetition loops.
21 July 2026 — mistral-small-3.2-24b. Dense 24B, faster and cheaper. Fixed the loops. Replaced after real use exposed terse output, absurd refusals and broken formatting.
22 July 2026 — gpt-oss-120b. Best tool-calling of the lineup, zero refusals, zero format failures. Current default.
24 July 2026 — the verification. 162 live queries across three models and two languages, on the real product stack, to check that the decision held up under measurement rather than impression.

Pick one: not enough active parameters to think with

We started with gemma-4-26b-a4b. On paper it looked like an efficient choice: a mixture-of-experts model, 26B total parameters, cheap to serve. The number that mattered was the one we underweighted — only about 4B parameters active per token.

A mixture-of-experts model routes each token through a small subset of its parameters. Total size buys you knowledge breadth; active size buys you reasoning at each step. Four billion active parameters is not enough to hold a plan together while selecting the right function out of 63, filling its arguments correctly, reading the result, and deciding what to call next.

The failure was not subtle. It failed at function calling, and during the synthesis phase — the part where it turns tool results into prose — it degenerated into repetition loops, restating the same clause until the answer was unusable. That is a known failure mode of open mixture-of-experts models under generation pressure, and we had picked a configuration especially exposed to it.

Pick two: the loops stopped, the answers got worse

Nineteen days later we moved to mistral-small-3.2-24b: a dense 24B model, faster and cheaper to run. Dense means every parameter participates in every token, so the reasoning-per-step problem went away. Combined with sampling controls — a low temperature and a frequency penalty that curbs the degeneration common to open models — the repetition loops stopped.

Then real use started producing a different set of complaints. The output was terse where a marketer needed explanation. It produced absurd refusals — declining questions it was perfectly capable of answering. And its formatting broke: the structured parts of an answer did not come back reliably clean.

None of that shows up on a leaderboard. All of it shows up the first time someone asks a real question about their own traffic and gets a two-line non-answer.

Pick three: gpt-oss-120b

The next day we switched to gpt-oss-120b — an open-weight model under the Apache 2.0 licence, 117B total parameters with 5.1B active per token, a 128k context window, native MXFP4 quantization that lets it fit a single 80GB GPU, and first-class vLLM support.

It was the best tool-caller of the lineup. Zero refusals. Zero format failures. And it delivered quality on par with a model that costs roughly four times more to run — which matters, because an analytics assistant that people actually use runs a lot of queries.

Being open-weight is not a detail either. We can pin an exact version, inspect it, red-team it, and reproduce a result months later. A closed API can change underneath you silently; you find out from a support ticket.

Then we checked ourselves

Two swaps in three weeks based on impressions is not a process, so we built a benchmark: 18 scenarios in Spanish and English, three passes, three models — 162 live queries — against the real assistant endpoint, the real 63-tool inventory and a real account's data, with ground truth computed live from the analytics database so every figure could be checked.

gpt-oss-120b passed 18 of 18 grounding and injection traps and got 144 of 144 verified facts right. Mistral Small 3.2 passed 9 of 18. Its characteristic failure was not invention: asked about a campaign that does not exist, it acknowledges the absence and then drifts into describing other real campaigns with figures nobody asked for. In English it also claimed a false incapability — "I don't have access to the necessary tools" — instead of explaining where the data actually ends.

Mistral was genuinely the fastest model in the run, with a median response around three seconds, and its cost came within a few percent of the winner. It still lost, because the gap in trap performance was wide enough that the confidence intervals stopped overlapping. Speed is not worth much if the answer quietly changes the question.

Why changing twice is the feature, not the bug

Every one of these swaps was cheap for the customer and invisible in the product: the model is selected by configuration, and the product code is identical across models. That is deliberate. If replacing a model is expensive, you stop replacing it, and you start defending the one you have.

The alternative to changing twice is not stability — it is not looking. A team that never changes its model in a year of a moving field is either exceptionally lucky or not measuring. We would rather tell you our first pick was wrong than pretend a lineage that had three entries had one.

So the honest summary is: we shipped a model that could not do the job, replaced it with one that did the job badly in ways only real use reveals, and landed on one that we then tried hard to break and could not. The next entry in this lineage will be published the same way.

The full decision record — the market audit, the verified public benchmarks and the complete bilingual run — is in our model-selection report. If you want the methodology instead of the verdict, read how we benchmark our own AI, or meet Seal AI for what the assistant actually does.