ChatGPT search struggles with news facts despite confident tone
OpenAI’s ChatGPT search tool is unpredictable and frequently inaccurate when identifying news sources, according to researchers at Columbia’s Tow Center. The bot gave partially or entirely incorrect responses to most requests while rarely admitting uncertainty. OpenAI pushed back, calling the study an atypical test and promising ongoing improvements.
ChatGPT’s search engine has a bad habit of hallucinating news sources, and it delivers those errors with absolute confidence.
According to a study from Columbia’s Tow Center for Digital Journalism, OpenAI’s recently launched search feature frequently misattributes quotes, hallucinates publishers, and sometimes even routes traffic to plagiarized scrapers instead of original outlets.
A failing grade on basic news retrieval
Researchers tested ChatGPT search by feeding it 200 specific quotes from 20 different publications, asking the bot to find the original source.
Out of 200 attempts, ChatGPT returned partially or entirely incorrect responses 153 times. That is an error rate of more than 75 percent on what should be routine quote lookup.
What makes the results worse is the chatbot’s tone.
Despite being wrong most of the time, ChatGPT only admitted it was unsure seven times across the entire study. In only seven instances did it use hedging words like "appears," "might," or flat-out admit that it could not locate the piece.
Even restrictions did not stop it from guessing. Forty of the tested quotes came from publishers that actively block OpenAI’s web crawler. Instead of explaining that it lacked access, ChatGPT confidently made up incorrect sources.
Even worse, having a corporate partnership did not help. ChatGPT struggled to correctly source quotes from outlets that already have commercial content deals with OpenAI.
Hallucinations and credit theft
The researchers highlighted several concerning examples of how these failures play out.
In one case, ChatGPT took a quote from a letter to the editor in the Orlando Sentinel and attributed it to a story in Time magazine.
In another instance, when asked about a quote from a New York Times piece on endangered whales, ChatGPT served up a link to a completely scraped, plagiarized clone site rather than the Times itself.
OpenAI pushed back in a statement to the Columbia Journalism Review, arguing that Tow Center withheld data and methodology, and claiming the study represented an "atypical test" of its search product while promising ongoing improvements.
Why it matters
If you were hoping to use ChatGPT Search as a research assistant or fact-checker, this study is a clear warning sign. The tool is simply not reliable for tracking provenance or verifying statements, and the friction of double-checking its confident hallucinations defeats the purpose of an AI shortcut.
For publishers, the findings are even more troubling. OpenAI has spent months convincing legacy media to sign data-licensing pacts, pitching its search engine as a traffic driver for credible news. But if ChatGPT cannot get licensing partners right and actively drives readers to scraping farms, those promises ring hollow.
Calling quote-lookup an "atypical test" feels like a weak defense from OpenAI. Reverse-searching an excerpt is table stakes for any web search tool. Until OpenAI fixes the bot’s stubborn refusal to say "I don't know," ChatGPT Search remains a risky tool for anything that requires factual accuracy.
Sources (1)
- theverge.com(opens in a new tab)Originaltheverge.com