GPQA: A Graduate-Level Google-Proof Q&A Benchmark
We present GPQA, a challenging dataset of 448 multiple-choice questions
written by domain experts in biology, physics, and chemistry. We ensure that
the questions are high-quality and extremely difficult: experts who have or are
pursuing PhDs in the corresponding domains reach 65% accuracy (74% when
discounting clear mistakes the experts identified in retrospect), while highly
skilled non-expert validators only reach 34% accuracy, despite spending on
average over 30 minutes with unrestricted access to the web (i.e., the
questions are "Google-proof"). The questions are also difficult for
state-of-the-art AI systems, with our strongest GPT-4 based baseline achieving
39% accuracy. If we are to use future AI systems to help us answer very hard
questions, for example, when developing new scientific knowledge, we need to
develop scalable oversight methods that enable humans to supervise their
outputs, which may be difficult even if the supervisors are themselves skilled
and knowledgeable. The difficulty of GPQA both for skilled non-experts and
frontier AI systems should enable realistic scalable oversight experiments,
which we hope can help devise ways for human experts to reliably get truthful
information from AI systems that surpass human capabilities.
655c17afe8a8971e89a60e2c
David Rein
extracted_confirmed
2024-06-24T02:49:26.645Z
655cc37f9af37bae32110d53
https://cdn-avatars.huggingface.co/v1/production/uploads/655cc37f9af37bae32110d53/cFGjZk0Y8gwDrwmRF83oO.jpeg
David Rein
user
Idavidrein
655c17afe8a8971e89a60e2d
Betty Li Hou
655c17afe8a8971e89a60e2e
Asa Cooper Stickland
admin_assigned
2023-11-21T13:58:38.252Z
6079eec3c5e1e416b294ea4f
/avatars/9672fb4c86ec5b20155e2cd941a4edf8.svg
Asa Cooper Stickland
user
Asa
655c17afe8a8971e89a60e2f
Jackson Petty
claimed_verified
2023-11-21T14:44:27.130Z
655cbcde95959f36af636d9a
https://cdn-avatars.huggingface.co/v1/production/uploads/655cbcde95959f36af636d9a/wV-pwUpUeDOjX9iUNa9Zq.jpeg
Jackson Petty
user
jowenpetty
655c17afe8a8971e89a60e30
Richard Yuanzhe Pang
claimed_verified
2023-11-21T14:44:24.216Z
64d50fa59a6a7ae984d90852
/avatars/26b78c151e707e3f9a9516c877d18ff7.svg
Richard Pang
user
yzpang
655c17afe8a8971e89a60e31
Julien Dirani
admin_assigned
2023-11-21T13:58:02.371Z
638a7bd8fb4c3732d06926ac
/avatars/33c5d04d524033249853670b7a74ac99.svg
Julien Dirani
user
jdirani
655c17afe8a8971e89a60e32
Julian Michael
655c17afe8a8971e89a60e33
Samuel R. Bowman
GPQA: A Graduate-Level Google-Proof Q&A Benchmark
We present GPQA, a challenging dataset of 448 multiple-choice questions
written by domain experts in biology, physics, and chemistry. We ensure that
the questions are high-quality and extremely difficult: experts who have or are
pursuing PhDs in the corresponding domains reach 65% accuracy (74% when
discounting clear mistakes the experts identified in retrospect), while highly
skilled non-expert validators only reach 34% accuracy, despite spending on
average over 30 minutes with unrestricted access to the web (i.e., the
questions are "Google-proof"). The questions are also difficult for
state-of-the-art AI systems, with our strongest GPT-4 based baseline achieving
39% accuracy. If we are to use future AI systems to help us answer very hard
questions, for example, when developing new scientific knowledge, we need to
develop scalable oversight methods that enable humans to supervise their
outputs, which may be difficult even if the supervisors are themselves skilled
and knowledgeable. The difficulty of GPQA both for skilled non-experts and
frontier AI systems should enable realistic scalable oversight experiments,
which we hope can help devise ways for human experts to reliably get truthful
information from AI systems that surpass human capabilities.
A dataset of extremely difficult multiple-choice questions challenges both experts and AI systems, facilitating the development of scalable oversight methods for AI-generated knowledge.
GPQA multiple-choice questions domain experts biology physics chemistry GPT-4 AI systems scalable oversight human supervision truthful information