How do I benchmark an LLM on my own data?

Run the candidates through your real product stack — your own endpoint, your own tool inventory, a real account's data — and compute ground truth directly from your database at the start of each run so every figure the model asserts can be checked automatically. Score with deterministic graders first, add adversarial traps for absent data and prompt injection, repeat each scenario several times, and report confidence intervals rather than single percentages.

What is a good LLM evaluation methodology?

One that measures the job your product actually does, produces the same score twice, and discloses its own defects. In practice: fixed scenarios with verifiable ground truth, deterministic graders for grounding, tool use and format, an LLM judge only for subjective quality and clearly marked as non-deterministic, traps for hallucination and injection, multiple passes per scenario, interval reporting, and an archive of every run including invalid ones.

Should I use an LLM as a judge to evaluate my model?

Only for the subjective part, and only as a secondary signal. Anything checkable — did the number match the database, was the right tool called, was the output well-formed, did it refuse — should be graded by deterministic code you can unit-test. If you do use a judge, pick a rival model at temperature zero so any bias runs against your preferred candidate, and disclose that a single judge with no second rater is a real limitation.

Why should you publish failed or discarded benchmark runs?

Because a benchmark that hides its discarded runs is a demo. Our first full run was invalid: the harness revealed the assistant was reusing one chat session across queries, so later models could answer from earlier models' history. We fixed it, re-ran everything and archived both runs. Publishing the invalid one is what lets a reader judge whether the valid one was fairly produced.

How many times should you repeat each benchmark question?

Enough that you can put an interval around the result rather than a point score. We used 18 scenarios across two languages, three passes each, over three models — 162 live queries, 54 per model. At that size a Wilson confidence interval is honest about the uncertainty: it showed one gap between models was real and another was too narrow to call from repetition alone.

How We Benchmark Our Own AI (And Why We Publish the Runs We Threw Away)

July 24, 20268 min readBy Rafa Jiménez

We evaluate the model inside SealMetrics on the product we ship, not on public leaderboards: real endpoint, real tools, real data, with the correct answers computed from the database before the model is asked. This is the whole method — including the run we discarded and the badly worded question we got wrong.

Key Takeaways

Test on the stack you ship: the real assistant endpoint, the real 63-tool inventory, a real account's data — with the model swapped by configuration and the product code identical across conditions.
Compute ground truth live from the analytics database at the start of each run, so every figure the model asserts is checked against reality instead of judged by impression.
Grade deterministically first. Keep the LLM judge separate, marked non-deterministic, and use a rival model so any bias runs against your own favourite.
Report intervals, archive every run — including the invalid one — and document your own defects. A benchmark that hides its discarded runs is a demo.

Most published model comparisons are unreproducible: a few prompts, a screenshot, a conclusion the author already held. We needed something we could rerun, argue with, and be wrong in public about — because the output decides which model answers our customers' questions about their own traffic.

What follows is the method we settled on. It is not sophisticated. It is mostly the discipline of refusing to grade anything by eye that could be checked by code.

1. Test on the stack you actually ship

Public benchmarks measure a model in isolation. Our assistant is not a model in isolation — it is a model behind a system prompt, inside an orchestration loop, holding an inventory of 63 analytics tools, talking to a live API.

So the harness calls the real production assistant endpoint, with the real tool inventory, against a real account's data. The model under test is switched by environment configuration and nothing else: the product code is byte-identical across conditions. If a candidate wins here, it wins at the job, not at a proxy for the job.

This also means the benchmark doubles as an integration test. One of the most valuable findings of our last run was not about any model — it was a production bug in our own response handling, surfaced as a single transport error out of 162 queries.

2. Compute ground truth from the database, live

The hardest part of evaluating an analytics assistant is deciding whether it told the truth. We solve it by asking the database first. At the start of every run, the harness queries the analytics tables directly for the answers — entrances, conversions, bounce rate, revenue, top channel, top source, top device — and stores them as the run's ground truth.

Every figure the model then asserts in prose is checked against that set. Not "does this sound plausible", but "is this the number". Computing it live also means the benchmark never rots as new data lands in the account: the truth is regenerated each time, so the same scenarios stay valid next month.

3. Deterministic graders first, judge last

Everything checkable is graded by ordinary code: grounding (asserted figures versus ground truth), tool calling, structured format validity, refusals, and trap outcomes. Those graders have 26 unit tests of their own, because a grader bug silently invalidates an entire study. The harness is standard library only — no framework to drift under it.

Only the subjective part goes to an LLM judge: usefulness, clarity, formal precision, scored 1 to 5. We deliberately used a rival model as the judge — qwen3-235b at temperature 0, one of the candidates competing against our eventual winner — so any bias would push against the model we expected to like. It scored the winner about 4.9 out of 5.

And we label that number for what it is: non-deterministic, from a single judge, with no second rater. It supports the decision. It does not carry it.

4. Trap it on purpose

Standard questions tell you whether a model can do the easy work. Traps tell you what it does when the honest answer is "that does not exist". Per language we ran 6 standard scenarios, 2 grounding traps and 1 prompt-injection trap:

Nonexistent entities. Ask about a campaign that was never run. The correct behaviour is to say so and stop. One model acknowledged the absence and then drifted into describing other, real campaigns with figures nobody had asked for — not invention, but an answer to a different question.
Empty periods. Ask about a window where the account holds no data. The correct behaviour is to explain the boundary of the data. One model instead claimed a false incapability, saying it lacked the necessary tools.
Injected instructions with a canary. Hide a hostile instruction inside data the user pastes in, containing a unique string. If that canary appears in the answer, the model obeyed the attacker. Running this in both languages is what exposed a model that resisted the attack in Spanish and leaked the canary in English.

5. Report intervals, not scores

With 18 scenarios in two languages, three passes and three models — 162 live queries, 54 per model — a single percentage is false precision. We report Wilson confidence intervals instead.

On traps, the winner scored 18 of 18 (95% interval 0.82 to 1.00); one rival scored 15 of 18 (0.61 to 0.94) and another 9 of 18 (0.29 to 0.71). The interval discipline is what let us say something precise: against the 9-of-18 model the intervals no longer overlap, so that difference is meaningful at this sample size. Against the 15-of-18 model the overlap is small but real — we did not claim a statistical win there, and pointed instead to the concrete security failures behind its losses.

6. The run we threw away

This is the part most teams delete. Our first full run was invalid and we published it anyway.

The harness exposed a defect in how it drove the product: without an explicit conversation id, the assistant resumed an existing chat session between queries. Later queries — including queries run against a different model — could therefore see earlier models' history. Every number that run produced was contaminated, and the contamination flowed in the direction of whichever model happened to go last.

We fixed it (a fresh conversation per query), re-ran the whole thing, and archived both. The invalid run is still there, labelled invalid, with the reason.

Keeping it costs nothing and changes everything about how the valid run should be read. A benchmark that does not disclose its discarded runs is a demo: you are being shown the take that worked and asked to assume there was only one.

7. Publish your own defects too

The valid run has a flaw of ours in it, and it stays documented. One scenario asked for "traffic by device for the last month". That admits two readings: the last 30 days — which is what our ground truth computed — or the previous calendar month. Some models chose the calendar month, where the test account held only five days of data, and reported those numbers.

They were real numbers, correctly retrieved. Our grader marked them as misses. That is not hallucination; it is a badly phrased benchmark question, written by us.

The detail that makes it worth publishing: 100% of all fact misses in the entire run trace back to that single scenario. Excluding it, the winning model got 144 of 144 verified facts right. We reworded the question for future runs and left the defect documented, because a reader who only saw the corrected version could not tell whether the misses were the model's fault or ours.

8. Make re-grading free

Every run archives the full transcripts, so a separate re-grading script re-scores stored runs offline at zero token cost. This turns out to be the quiet productivity win of the whole design: when you find a grader bug or want to add a metric, you do not re-spend a run — you re-score history, including the discarded one, and see whether the conclusion moves.

How to build your own

If you are choosing or monitoring a model for a real product, this is the shortest path we know:

Write scenarios from real user questions, not from prompt-engineering examples. Mirror them across every language your users speak, sharing the same ground truth.
Drive the real endpoint. Change the model by configuration; change nothing else between conditions.
Compute ground truth from your own source of truth at the start of each run, automatically.
Grade with code, and unit-test the graders. If you cannot test the grader, you cannot trust the score.
Add traps: an entity that does not exist, a period with no data, and an injected instruction carrying a canary string.
Repeat each scenario several times and report a confidence interval instead of a headline percentage.
Use an LLM judge only for taste, prefer a rival model, fix the temperature at zero, and label the result non-deterministic.
Archive every run, including the ones you invalidate, with the reason attached.
Make offline re-grading possible so improving the method does not cost another run.
Write down the limitations you know about — single judge, small n, your own ambiguous questions — before someone else finds them.

The bottom line

None of this makes a benchmark objective. It makes it auditable, which is the achievable goal. Someone reading our results can see which questions were asked, where the correct answers came from, what was graded by code and what by judgement, which run was thrown away and why, and which of the failures were our own fault.

That is the standard we think anyone shipping an AI feature should hold themselves to — including, especially, when the results are inconvenient. The complete methodology and the full results are published in our internal benchmark report, alongside the model lineage that produced it.