Grounding: Why a Good Analytics AI Shouldn't 'Know' Anything

There is a product principle behind AI analytics that sounds wrong the first time you hear it: the model should not know anything.

Not "should not know much". Should not rely on knowing. Every fact it states about your business should have arrived in the last few hundred milliseconds, from your database, through a tool call — not from a weight matrix trained months ago on the public internet.

That constraint has a name. It is called grounding, and it is the difference between an assistant you can put in front of a marketing team and a very confident text generator.

Knowing versus looking up

A general-purpose chatbot answers from parameters. You ask it the capital of Peru and it produces "Lima" because that association is encoded in its weights. There is no lookup, no source, no citation — just a very good statistical guess. Most of the time it is right. When it is wrong, it is wrong in exactly the same tone of voice.

An analytics assistant cannot work that way, for an obvious reason: your entrances last Tuesday were never in anyone's training data. There is nothing to recall. If the model produces a number, it either fetched it or fabricated it. There is no third option.

So the design goal flips. Instead of making the model smarter about the world, you make it structurally incapable of answering without first going to get the data.

What grounding actually looks like in the stack

Grounding is not a line in the system prompt saying "do not make things up". Models agree to that instruction and then make things up anyway. It is a shape you give the whole request path:

1. The question arrives in natural language. "Which channel drove the most conversions last month?"
2. The model selects tools, not answers. Seal AI's assistant has a 63-tool inventory — overview, channels, campaigns, funnels, segments, landing pages, and so on. Its first job is to decide which of them to call and with what parameters.
3. The tools query your data. Real rows, real aggregates, scoped to your account and your timezone.
4. The results travel back into the prompt. The numbers are now literally in front of the model as text.
5. The model narrates and interprets. It reads what came back, arranges it, notices the trend, suggests what to look at next. It is doing language work over evidence, not retrieval from memory.

Step five is the only step where the model's intelligence is spent on the actual question. Everything before it is plumbing — and the plumbing is what makes the answer true.

The payoff: a "bad" model can be the right model

Here is the honest version of our own case. The model inside Seal AI is gpt-oss-120b. On SimpleQA — a benchmark of short open-world factual questions — it scores 0.168 accuracy with a 0.782 hallucination rate. Read cold, that looks disqualifying. A model that confidently invents answers to four out of five general-knowledge questions is not something you would want narrating a revenue report.

Except it is never asked a general-knowledge question. Nobody opens an analytics tool to find out who won a 1994 election. The assistant is asked to read entrances, conversions, channels and bounce rates that arrived in its context window a moment ago, and to explain what they mean. The capability being measured by SimpleQA is one the product deliberately does not use.

What the product does use, the same model is strong at: MMLU 90.0, MMLU-Pro 80.8, GPQA Diamond 80.1 without tools, and — most relevant here — reliable function calling against a large tool inventory. Grounding is what lets you spend your model budget on the capabilities that matter and ignore the ones that don't.

Grounding you trust is grounding you tested

An architecture that should prevent invention is not the same as one that does. Models can call a tool, get a partial answer, and then quietly fill the gap with something reasonable. The only way to know is to check every number.

That is how we ran our own internal benchmark: 18 scenarios in two languages, three passes, three candidate models — 162 live queries against the real production assistant endpoint with the real 63-tool inventory. Before each run the harness computed ground truth directly from the analytics database — entrances, conversions, bounce rate, revenue, top channel, top source, top device — so every figure the assistant asserted could be compared to reality by a deterministic grader, not by a human reading along and nodding.

The shipped model asserted 144 of 144 verified facts correctly. Every single fact miss in the entire run traced back to one badly phrased question of ours, which we will come back to.

Traps: asking about things that do not exist

Checking correct answers only tells you half the story. The interesting failure is not a wrong number — it is a number where there should have been an admission.

So each language block included grounding traps: questions about a campaign that was never run, or a period for which the account holds no data. There is exactly one right behaviour — say the data is not there, and say why. Anything fluent and numeric is a failure, no matter how well written.

The results were more interesting than a pass/fail table. The shipped model passed 18 of 18 traps (95% Wilson confidence interval 0.82-1.00). One alternative, mistral-small-3.2, passed 9 of 18 — and its failure mode was subtle. Asked about a nonexistent campaign, it correctly acknowledged the absence, then drifted into describing other real campaigns with figures nobody asked for. That is not invention; the numbers were true. It is answering a different question, which in a report is its own kind of wrong. In English it also claimed a false incapability — "I don't have access to the necessary tools" — instead of explaining the data's time boundary.

The Wilson intervals matter here: the winner's interval no longer overlaps mistral's, so at this sample size that gap is statistically meaningful rather than a lucky run.

The question we got wrong

One scenario asked for "traffic by device for the last month". That admits two readings: the last 30 days, which was our ground truth, or the previous calendar month. Some models chose the calendar month, where the test account held only five days of data, and answered with those real numbers. The grader marked them as misses.

They were not hallucinations. They were correct answers to a question we had phrased badly. One hundred per cent of the fact misses in the whole run trace to that single scenario. We reworded it, and we keep the defect documented, because a benchmark that only reports its flattering results is a demo. The same applies to our first full run, which we discarded entirely: the harness revealed that the assistant was reusing the chat session between queries, letting later models read earlier models' history. We fixed it, re-ran, and archived both.

What this means when you use the assistant

Practically: ask it why. A grounded assistant can tell you which metric, which period and which breakdown produced a figure, because it fetched them seconds ago. If an assistant cannot show you its evidence, the number is a claim, not a measurement.

And test it once with something that does not exist — a campaign you never ran, a month before you installed the tracker. The answer you want is the boring one: there is no data for that. An assistant willing to say "I don't have that" is the only kind worth believing when it says it does.

More on how the model was chosen in the write-up of the 162-query benchmark, and on the architecture behind it in the Seal AI architecture documentation.