How to fact check a document with ChatGPT, and what it cannot do

Why it cannot verify a claim

Paste in a document, ask for a fact check, and you get a tidy list of statements marked correct or questionable. Nothing was consulted to produce it.

The model is generating the text that a fact check usually looks like. Where a claim is common and well represented, that text is often right. Where a claim is specific, recent, contested or private to your organisation, the same fluent format arrives with nothing behind it. Both cases read identically, which is the actual problem. You cannot sort the good verdicts from the bad ones without doing the checking you were trying to avoid.

This is starkest with internal documents. If your board pack says the Manchester site ran at 71 percent utilisation in Q2, there is no public fact of the matter. Any judgement about that number is invention. The only real check is the system the figure came from.

What it is genuinely good at

Triage. It reads a 90 page report faster than you can, and it reliably recognises the grammatical shape of a claim: an attribution, a statistic, a causal assertion, a superlative.

That is the expensive part of fact checking. Finding the forty claims in a long document that carry weight, separating them from the sixty that are framing, and ordering them by what breaks if they are wrong. Verification is then a short focused job on a short list rather than an open ended read.

Sort claims by what would settle them

The useful question is never "is this true". It is "what would I have to open to find out".

Ask for the classification explicitly and the output stops pretending. A claim it marks as needing the original report is more honest, and more useful, than a claim it marks as true.

The prompt sequence that works

1. Extract, do not judge

   "List every factual claim in this document. For each one give the exact sentence, the section or page, and the type: statistic, attribution, causal claim, prediction, definition. Do not say whether any of them are true."

2. Classify by how it could be checked

   "For each claim, say what a person would need to open to verify it: a named external source, a calculation inside this document, an internal system, or nothing available."

3. Rank by consequence

   "Which five of these, if wrong, would change the conclusion of the document or embarrass whoever published it?" This is the list you spend your afternoon on.

4. Ask for the internal contradictions separately

   "Find places where this document contradicts itself: conflicting figures for the same measure, dates that cannot both be right, totals that do not match their components, definitions that change partway through."

Give it the source and the check becomes possible

Verification turns into comparison the moment both documents are present.

Upload the underlying report alongside the document that cites it and ask a narrow question: does the document represent this source correctly, and does the figure on page 4 appear in the source at all. That is a text comparison, which is work it can do. It is also most of what practical fact checking consists of, because the common failure is not fabrication but a figure quoted out of context, a projection reported as an outcome, or a hedged finding repeated as settled.

Keep each check to one source and one document. Three sources at once and attribution blurs, at which point you are back to trusting the summary.

Where it will mislead you

Fabricated citations. Corrections arrive with references because references are part of the pattern. Open every one.

Confident false negatives. It will occasionally flag a correct statement as dubious, which costs you time and, worse, teaches you to discount its flags.

Silent truncation. Text and document files are capped at 2 million tokens, and past that the tail is dropped without a warning in the answer. A clean fact check of a document it half read is the most dangerous output on this page.

Charts and figures. Document retrieval is text only, except Enterprise, so images inside the file are discarded and a graph with a misleading axis is simply not there to question. Screenshot it and upload it as an image if the figures matter.

The practical decision

Ask it what to check. Do not ask it what is true.

An hour spent turning a long document into a ranked list of twelve claims, each tagged with the source that would settle it, is an hour that makes the checking tractable. An hour spent reading its verdicts is an hour that produces a document you now believe for no reason.