Checking the Sources an AI Research Tool Cites

Five cited links, two of them wrong. That was one research answer on a client question, and it is the reason I stopped treating a list of sources as a finished job.

What I found when I checked

I use both for client questions: ChatGPT on the free account, and Claude Pro with web search on. On one client answer from Claude I opened all 5 of the links it cited, and 2 of them did not hold up. One page did not contain the claim at all. Another quoted a price the vendor had already changed. Nothing in the answer looked wrong, the sentences were calm and the links were real links, which is why I now open every one before I put anything in front of a client.

What a citation actually carries

Anthropic's documentation for its web search tool explains what gets attached to an answer. Each citation carries the source URL, its title, and up to 150 characters of the cited content. Search results also carry a page_age field, described as when the site was last updated.

So the link is the address of a page that was in the search results. The quoted fragment is short, and it is the part the model saw when it wrote the sentence. Reading that fragment tells you the sentence had a basis. It does not tell you the page still says that today, or that the fragment meant what it appears to mean in the middle of the page it came from.

The two ways it goes wrong on real work

The first is a link that points at the right site and the wrong fact. A vendor's documentation covers twenty features on one long page, the model attaches the page, and the specific claim it made is nowhere on it. That one is easy to catch if you open the link and search the page for the number.

The second is worse for the kind of articles I write, because the page is correct and old. A price from last year sits on a URL that has not changed. The answer repeats it in the present tense, and unless you know the vendor moved its prices, nothing in the answer looks stale. A date on the source is the only defence I have found, and the date has to be the day I actually read it.

What I do now

For anything a client will see, I open every cited link and search the page for the exact number or phrase. If the claim is a price or a limit, I take it from the vendor's own page and write down the date I read it. If a link does not load, the claim goes out of the draft rather than in with a hedge.

That sounds slow for 5 links. It is faster than explaining to a client why a number in my document is a year out of date. I keep the two assistants for different jobs, which I wrote about in ChatGPT vs Claude for coding, and I treat both the same way on anything checkable.