UQ: Assessing Language Models on Unsolved Questions
Benchmarks shape progress in AI research. A useful benchmark should be both
difficult and realistic: questions should challenge frontier models while also
reflecting real-world usage. Yet, current paradigms face a difficulty-realism
tension: exam-style benchmarks are often made artificially difficult with
limited real-world value, while benchmarks based on real user interaction often
skew toward easy, high-frequency problems. In this work, we explore a radically
different paradigm: assessing models on unsolved questions. Rather than a
static benchmark scored once, we curate unsolved questions and evaluate models
asynchronously over time with validator-assisted screening and community
verification. We introduce UQ, a testbed of 500 challenging, diverse questions
sourced from Stack Exchange, spanning topics from CS theory and math to sci-fi
and history, probing capabilities including reasoning, factuality, and
browsing. UQ is difficult and realistic by construction: unsolved questions are
often hard and naturally arise when humans seek answers, thus solving them
yields direct real-world value. Our contributions are threefold: (1) UQ-Dataset
and its collection pipeline combining rule-based filters, LLM judges, and human
review to ensure question quality (e.g., well-defined and difficult); (2)
UQ-Validators, compound validation strategies that leverage the
generator-validator gap to provide evaluation signals and pre-screen candidate
solutions for human review; and (3) UQ-Platform, an open platform where experts
collectively verify questions and solutions. The top model passes UQ-validation
on only 15% of questions, and preliminary human verification has already
identified correct answers among those that passed. UQ charts a path for
evaluating frontier models on real-world, open-ended challenges, where success
pushes the frontier of human knowledge. We release UQ at
https://uq.stanford.edu.
68ad09b186b21a0e2e358cf5
Fan Nie
claimed_verified
2025-08-27T07:11:28.545Z
652fb9e437fad005c70d66f4
https://cdn-avatars.huggingface.co/v1/production/uploads/652fb9e437fad005c70d66f4/xjvro1bFvQ8ILR0tkD6qU.jpeg
Fan Nie
user
FanNie1208
68ad09b186b21a0e2e358cf6
Ken Ziyu Liu
claimed_verified
2025-08-26T09:50:32.267Z
64f135ed1e05348eda3edd33
/avatars/69f9d17711c3c619c14e1dce53cebd9d.svg
Ken Liu
user
kzliu
68ad09b186b21a0e2e358cf7
Zihao Wang
68ad09b186b21a0e2e358cf8
Rui Sun
68ad09b186b21a0e2e358cf9
Wei Liu
68ad09b186b21a0e2e358cfa
Weijia Shi
68ad09b186b21a0e2e358cfb
Huaxiu Yao
68ad09b186b21a0e2e358cfc
Linjun Zhang
68ad09b186b21a0e2e358cfd
Andrew Y. Ng
68ad09b186b21a0e2e358cfe
James Zou
68ad09b186b21a0e2e358cff
Sanmi Koyejo
68ad09b186b21a0e2e358d00
Yejin Choi
68ad09b186b21a0e2e358d01
Percy Liang
68ad09b186b21a0e2e358d02
Niklas Muennighoff
UQ: Assessing Language Models on Unsolved Questions
Benchmarks shape progress in AI research. A useful benchmark should be both
difficult and realistic: questions should challenge frontier models while also
reflecting real-world usage. Yet, current paradigms face a difficulty-realism
tension: exam-style benchmarks are often made artificially difficult with
limited real-world value, while benchmarks based on real user interaction often
skew toward easy, high-frequency problems. In this work, we explore a radically
different paradigm: assessing models on unsolved questions. Rather than a
static benchmark scored once, we curate unsolved questions and evaluate models
asynchronously over time with validator-assisted screening and community
verification. We introduce UQ, a testbed of 500 challenging, diverse questions
sourced from Stack Exchange, spanning topics from CS theory and math to sci-fi
and history, probing capabilities including reasoning, factuality, and
browsing. UQ is difficult and realistic by construction: unsolved questions are
often hard and naturally arise when humans seek answers, thus solving them
yields direct real-world value. Our contributions are threefold: (1) UQ-Dataset
and its collection pipeline combining rule-based filters, LLM judges, and human
review to ensure question quality (e.g., well-defined and difficult); (2)
UQ-Validators, compound validation strategies that leverage the
generator-validator gap to provide evaluation signals and pre-screen candidate
solutions for human review; and (3) UQ-Platform, an open platform where experts
collectively verify questions and solutions. The top model passes UQ-validation
on only 15% of questions, and preliminary human verification has already
identified correct answers among those that passed. UQ charts a path for
evaluating frontier models on real-world, open-ended challenges, where success
pushes the frontier of human knowledge. We release UQ at
https://uq.stanford.edu.
UQ is a benchmark for evaluating AI models on unsolved questions, combining difficulty and realism to assess capabilities like reasoning, factuality, and browsing.
unsolved questions UQ UQ-Dataset UQ-Validators UQ-Platform generator-validator gap human verification reasoning factuality browsing