Why do LLMs return malformed structured output?

Because generation is probabilistic and your schema is not. A model that has understood the request perfectly can still express it in a shape your contract does not allow — for example emitting a list where the schema expects a string, because the question implied two data series instead of one. It is usually not a comprehension failure; it is a serialisation choice that your validator was never told to accept.

How do I validate structured output from an LLM?

Validate at the parse boundary, and decide per field what a failure means. Coerce malformations you can recover unambiguously, drop the elements you cannot and log a warning, and reject the whole response only when the part that failed is the part the user actually asked for. Treat every field as untrusted input, exactly as you would treat a request body from the public internet.

Should a malformed chart make an API request fail?

No. If the text answer is already generated and correct, a decorative element that fails validation should degrade — not take the response with it. Our bug did exactly the wrong thing: a chart whose axis key had the wrong type caused the whole chat response to return HTTP 500, so the user lost a perfectly good answer over a visual they did not ask for.

How do you catch LLM bugs before users report them?

Run your own product as a test subject at volume. We found this one as a single transport error in 162 live benchmark queries against the real production endpoint — a rate low enough that manual testing would likely have missed it and high enough that users would eventually have hit it. Benchmarks that drive the real stack, not a mock, double as integration tests.

What is a good fallback when LLM output fails schema validation?

A two-layer one. First, a coercion layer at the parse boundary that repairs known-recoverable shapes so the element survives. Second, an airbag that drops any element still invalid after coercion, logs a warning, and lets the rest of the response through. Then add a regression test for every malformation you have actually seen, because the same shape will come back.

Our AI Got It Wrong in Production — And Our Own Test Caught It

July 24, 20266 min readBy Rafa Jiménez

Someone asked for a comparison. The model answered correctly, then attached a chart whose y-axis key was a list instead of a string. Schema validation rejected the chart, and the entire response failed with a 500 — a good answer thrown away because of a decoration.

Key Takeaways

A comparison question made the model emit two data series in a field the schema declared as a single string. Strict validation rejected it and the whole chat response returned HTTP 500.
We found it as 1 transport error in 162 live benchmark queries — rare enough to survive manual testing, common enough to reach users.
The fix has two layers: coerce recoverable malformations at the parse boundary, and an airbag that drops any still-invalid element with a warning rather than failing the response.
The general rule: treat every field of LLM structured output as untrusted input, and never let a cosmetic element decide whether the user gets an answer.

We build Seal AI on the assumption that we will be the ones to find its bugs. This is one we found, in the product as shipped, and it is a better teacher than any success story we could publish.

What happened

The assistant answers analytics questions in plain language, and when a chart helps, it returns one alongside the text — a structured object describing what to plot, validated against a strict schema before it reaches the browser.

Asked for a comparison, the model did something entirely reasonable from its point of view: it produced a chart with two series. To express that, it set the y-axis key to a list of two field names. Our schema said that key is a string.

Validation rejected the object. The rejection propagated. The whole chat response returned HTTP 500.

That is the part worth sitting with. The text answer had already been generated. It was correct, grounded in real data, and useful. The user got none of it — because a decorative element failed a type check.

How we found it

It surfaced as a single transport error in the 162 live queries of our internal benchmark, which drives the real production endpoint rather than a mock. One error in 162 is an awkward frequency: too rare for anyone to reproduce it by hand during a review, too common to stay theoretical once a product answers thousands of questions a week.

We were running that benchmark to compare models. We got a production defect out of it instead — which is the strongest argument we know for benchmarking against your real stack instead of a simulation. A mock would have validated our schema against our own assumptions and passed.

Why this class of bug is inevitable

It is tempting to file this as "the model got it wrong". It did not, really. It understood the request — a comparison needs two series — and expressed that understanding in a shape our contract had not anticipated.

That is the permanent condition of LLM structured output: a probabilistic generator on one side, a strict schema on the other. Prompting reduces the mismatch rate. It never reaches zero, especially at the edges — comparisons, multi-metric requests, unusual periods, other languages. If your architecture assumes the rate is zero, you have not built a feature, you have built a coin-flip with good odds.

Our mistake was not the schema being strict. Strict is correct. Our mistake was giving a strict validator the authority to fail an entire response over an optional element.

The two-layer fix

We did not patch the prompt and call it done. Prompt tweaks are not a boundary; they are a hope. We changed the boundary itself.

Coercion at the parse boundary. When the y-axis key arrives as a list, we take its first series and continue. The malformation is unambiguously recoverable, so the chart survives — degraded to one series rather than lost entirely, which is a far better outcome than nothing.
An airbag behind it. Any chart still invalid after coercion is dropped, with a warning logged, and the response continues without it. One bad chart can no longer take down an answer. Ever.

We then applied the same shape of hardening to the conversation-history load path, where a single malformed stored element could equally have poisoned the load of an entire past conversation. And we added regression tests for the exact malformation, because the same shape will come back — models change, and this one was a sensible thing to emit.

Three rules for anyone shipping LLM structured output

Treat every field as untrusted input. Model output is not your own data structure round-tripping. It is a payload from a system you do not control, and it deserves the same scepticism as a request body arriving from the public internet — type checks, bounds, and an explicit decision about what happens when it is wrong.
Grade elements by how much the user needs them. Decide, per element, whether a validation failure is fatal. The answer the user asked for is fatal. A chart, a suggested follow-up, an icon, a highlight — none of those are. Failing the whole response over an optional part converts a cosmetic glitch into an outage.
Recover what you can, drop what you cannot, log everything. Coerce the malformations with an obvious correct reading, drop the rest with a warning so you keep visibility, and write a regression test for every shape you have actually observed in the wild. Silent recovery with no logging is how you stop learning from your own model.

Why we are telling you

Because "our AI works" is not a claim anybody should accept on trust, including from us. What is checkable is whether a vendor is looking — whether they run their own product hard enough to find the one-in-a-hundred failure, and whether they say what they found.

This one cost a user-visible response in a test. Fixed, it cost a coercion function, an airbag, and a handful of regression tests. Undetected, it would have been an intermittent 500 that nobody could reproduce, on a feature people were starting to rely on.

The benchmark that caught it, including its methodology and the runs we discarded, is published in our internal benchmark report.