OpenAI's SimpleQA puts AI honesty to the test
OpenAI just open-sourced SimpleQA, a factual accuracy benchmark packed with 4,326 short, single-answer questions built to expose hallucinations. Frontier models like GPT-4o struggle hard, scoring under 40% correct. Why it matters: Measuring how often models fabricate answers—and how persistently they overstate their confidence—is the first concrete step toward fixing AI's chronic guessing problem.
SimpleQA is an open-source factuality benchmark from OpenAI designed to do one specific job: measure how often frontier language models hallucinate when asked for cold, hard facts.
It consists of 4,326 short, fact-seeking questions spanning science, pop culture, video games, and history. Every single question has a distinct, indisputable target answer that does not change over time.
How it was built
To ensure the benchmark didn't grade models on bogus data, OpenAI hired human trainers to write questions specifically tuned to trigger hallucinations in GPT-4o and GPT-3.5.
Every question was then independently answered by a second trainer without seeing the original response, and only questions where both humans agreed made the cut. A third trainer audited a random sample of 1,000 questions with a 94.4% agreement rate, leaving the dataset's inherent error rate at roughly 3%.
Running SimpleQA is intentionally frictionless. Because the questions and ground-truth answers are short, you can use a prompted frontier model API like ChatGPT as an automated grader. Responses get sorted into three simple buckets: correct, incorrect, or not attempted.
How the models held up
Frontier models performed surprisingly poorly. Tested without web browsing, GPT-4o scored under 40% correct. Smaller models like gpt-4o-mini scored even lower due to less broad world knowledge.
Reasoning models showed a telling behavioral shift. Both o1-preview and o1-mini opted to not attempt questions far more often than GPT-4o. Their extra thinking time appeared to help them recognize when they simply didn't know the answer, dodging unnecessary hallucinations.
OpenAI also used SimpleQA to test calibration—whether models know what they actually know. Asking models to state their confidence revealed that while confidence correlates with accuracy, models consistently overstate how sure they are. Testing response consistency over 100 repeated runs showed o1-preview was the best-calibrated model in the lineup.
The benchmark code and dataset are public on GitHub. Keep in mind that SimpleQA only tests isolated, short-form facts; it does not yet evaluate long-form generation with dense factual claims.
Why it matters
If you build RAG pipelines, fine-tune models, or evaluate enterprise LLMs, SimpleQA is worth running right away. Older QA benchmarks like TriviaQA are saturated and practically useless for frontier models, while SimpleQA exposes exactly where a model's raw internal memory falls apart.
More importantly, it pushes the industry to reward models that admit ignorance rather than bluffing. Shifting the evaluation from smooth talk to precise factual calibration is essential if agentic AI is ever going to be trusted with real-world tasks.
Sources (1)
- openai.com(opens in a new tab)Originalopenai.com