How I stop an LLM from inventing facts
Thursday, September 25, 2026
Do not let the model write to the page. Let it propose, then compare every number in the output against the input as a multiset. If they do not match, reject the whole page. That catches invented figures. It does not catch changed meaning, and changed meaning is the one that gets you.
I let a model rewrite a law firm's website. Twice.
The firm is my partner's. The pages state Lebanese company law. Capital minimums, tax rates, article numbers.
Getting one wrong is not a typo. It is a lawyer publishing something false.
So the model never writes to the page. It writes a proposal. A guard decides.
The guard
Every number in the output is compared against the input. As a multiset, not a set. Two mentions of 17% are not one.
If they do not match, the whole page is rejected. Not the paragraph. The page.
That sounds heavy-handed. It is the point. A half-applied rewrite is harder to review than either version, because you no longer know which half you are reading.
The first rejection was mine
It threw out the offshore guide. Six invented numbers and a changed citation.
Both were my fault.
My HTML-to-markdown emitter wrote 1. for every ordered list item. The model counted properly. Six list items, six numbers that were not in the source.
And I had normalised art. 7 to Article 7 before comparing.
Neither was a fact changing. The guard was wrong, not the model.
What the guard cannot see
One paragraph came back like this.
It may serve clients anywhere.
It serves clients anywhere.
Same numbers. Same tags. Same length.
A permission turned into a habit, on a page about what a company is allowed to do.
Another one. An auditor is mandatory above a capital threshold, or if partners holding a fifth ask for one. Two triggers, both mandatory.
It came back as “partners may ask for one.” A trigger turned into a right.
I caught both by reading. There is no check for it.
The worst one
The formation sites are operated by a software company. The legal work is done by the firm. Two companies, on purpose.
I asked the model to write the terms. It came back in the first person.
We do the law.
Published by a software company. In the one document a regulator would quote back.
Now the guard rejects we, us, our and ours outright.
The first rerun failed seven of ten sections on it. That is not a near miss. That is a model refusing to hold a constraint.
Repeating the rule in the user prompt, not just the system prompt, fixed it.
A guard that cries wolf gets ignored
Mine rejected “prices are shown in US dollars.” It was matching us case-insensitively.
I made it case-sensitive the same hour.
If I had left it, I would have started skimming the rejections. Skimming is how the real one gets through.
What I actually do now
Guards catch the shape of error you thought of. Numbers, markup, pronouns. Not meaning.
So I still read every diff. Across four sites, 49 rewrites landed. I reverted three by hand.
The model is a fast writer with no stake in being right. The guard is cheap. The reading is the job.