SimpleQA Verified: A Reliable Factuality Benchmark to Measure Parametric
  Knowledge
We introduce SimpleQA Verified, a 1,000-prompt benchmark for evaluating Large
Language Model (LLM) short-form factuality based on OpenAI's SimpleQA. It
addresses critical limitations in OpenAI's benchmark, including noisy and
incorrect labels, topical biases, and question redundancy. SimpleQA Verified
was created through a rigorous multi-stage filtering process involving
de-duplication, topic balancing, and source reconciliation to produce a more
reliable and challenging evaluation set, alongside improvements in the
autorater prompt. On this new benchmark, Gemini 2.5 Pro achieves a
state-of-the-art F1-score of 55.6, outperforming other frontier models,
including GPT-5. This work provides the research community with a
higher-fidelity tool to track genuine progress in parametric model factuality
and to mitigate hallucinations. The benchmark dataset, evaluation code, and
leaderboard are available at:
https://www.kaggle.com/benchmarks/deepmind/simpleqa-verified.
68c0db913912ed54cf543246
Lukas Haas
68c0db913912ed54cf543247
Gal Yona
68c0db913912ed54cf543248
Giovanni D'Antonio
68c0db913912ed54cf543249
Sasha Goldshtein
68c0db913912ed54cf54324a
Dipanjan Das
SimpleQA Verified: A Reliable Factuality Benchmark to Measure Parametric
  Knowledge
We introduce SimpleQA Verified, a 1,000-prompt benchmark for evaluating Large
Language Model (LLM) short-form factuality based on OpenAI's SimpleQA. It
addresses critical limitations in OpenAI's benchmark, including noisy and
incorrect labels, topical biases, and question redundancy. SimpleQA Verified
was created through a rigorous multi-stage filtering process involving
de-duplication, topic balancing, and source reconciliation to produce a more
reliable and challenging evaluation set, alongside improvements in the
autorater prompt. On this new benchmark, Gemini 2.5 Pro achieves a
state-of-the-art F1-score of 55.6, outperforming other frontier models,
including GPT-5. This work provides the research community with a
higher-fidelity tool to track genuine progress in parametric model factuality
and to mitigate hallucinations. The benchmark dataset, evaluation code, and
leaderboard are available at:
https://www.kaggle.com/benchmarks/deepmind/simpleqa-verified.
SimpleQA Verified is a refined benchmark for evaluating the factuality of Large Language Models, addressing issues in previous benchmarks and providing a more reliable evaluation tool.
Large Language Model LLM SimpleQA SimpleQA Verified F1-score parametric model factuality hallucinations