ARB: Advanced Reasoning Benchmark for Large Language Models
Large Language Models (LLMs) have demonstrated remarkable performance on
various quantitative reasoning and knowledge benchmarks. However, many of these
benchmarks are losing utility as LLMs get increasingly high scores, despite not
yet reaching expert performance in these domains. We introduce ARB, a novel
benchmark composed of advanced reasoning problems in multiple fields. ARB
presents a more challenging test than prior benchmarks, featuring problems in
mathematics, physics, biology, chemistry, and law. As a subset of ARB, we
introduce a challenging set of math and physics problems which require advanced
symbolic reasoning and domain knowledge. We evaluate recent models such as
GPT-4 and Claude on ARB and demonstrate that current models score well below
50% on more demanding tasks. In order to improve both automatic and assisted
evaluation capabilities, we introduce a rubric-based evaluation approach,
allowing GPT-4 to score its own intermediate reasoning steps. Further, we
conduct a human evaluation of the symbolic subset of ARB, finding promising
agreement between annotators and GPT-4 rubric evaluation scores.
64c0846c6cf8e2b9f041d526
Tomohiro Sawada
claimed_verified
2023-07-27T08:49:18.370Z
63750b55a17c01a551cf41df
/avatars/f9d72caa51d2cb12a6e3e6f700ad35d9.svg
Tom Sawada
user
tsawada
64c0846c6cf8e2b9f041d527
Daniel Paleka
admin_assigned
2023-07-26T08:58:46.537Z
62363277cd8e1f47c865190a
/avatars/fb42d42dedd4dde8df4109883cbe7ea9.svg
Daniel Paleka
user
dpaleka
64c0846c6cf8e2b9f041d528
Alexander Havrilla
64c0846c6cf8e2b9f041d529
Pranav Tadepalli
64c0846c6cf8e2b9f041d52a
Paula Vidas
64c0846c6cf8e2b9f041d52b
Alexander Kranias
64c0846c6cf8e2b9f041d52c
John J. Nay
admin_assigned
2023-07-26T08:59:35.647Z
63f2b49f7ddf724fbcc534fd
/avatars/4076a12ac9423ab276d06334c7e392db.svg
John Nay
user
johnjnay
64c0846c6cf8e2b9f041d52d
Kshitij Gupta
claimed_verified
2024-03-14T08:00:36.366Z
625daa0c943e346492b63d21
/avatars/860f8e9bff85325096b6f8e15102def6.svg
Kshitij Gupta
user
kshitijkg
64c0846c6cf8e2b9f041d52e
Aran Komatsuzaki
admin_assigned
2023-07-26T09:03:20.918Z
5f35f6fa7e583543386216c9
https://cdn-avatars.huggingface.co/v1/production/uploads/5f35f6fa7e583543386216c9/7GR8PX6ahT-2dZdMZYz7T.jpeg
Aran Komatsuzaki
user
AranKomatsuzaki
ARB: Advanced Reasoning Benchmark for Large Language Models
Large Language Models (LLMs) have demonstrated remarkable performance on
various quantitative reasoning and knowledge benchmarks. However, many of these
benchmarks are losing utility as LLMs get increasingly high scores, despite not
yet reaching expert performance in these domains. We introduce ARB, a novel
benchmark composed of advanced reasoning problems in multiple fields. ARB
presents a more challenging test than prior benchmarks, featuring problems in
mathematics, physics, biology, chemistry, and law. As a subset of ARB, we
introduce a challenging set of math and physics problems which require advanced
symbolic reasoning and domain knowledge. We evaluate recent models such as
GPT-4 and Claude on ARB and demonstrate that current models score well below
50% on more demanding tasks. In order to improve both automatic and assisted
evaluation capabilities, we introduce a rubric-based evaluation approach,
allowing GPT-4 to score its own intermediate reasoning steps. Further, we
conduct a human evaluation of the symbolic subset of ARB, finding promising
agreement between annotators and GPT-4 rubric evaluation scores.
ARB is a novel benchmark featuring advanced reasoning problems across multiple domains, demonstrating that current LLMs score poorly on more challenging tasks and introducing a rubric-based evaluation approach.
Large Language Models (LLMs) ARB advanced reasoning problems mathematics physics biology chemistry law symbolic reasoning GPT-4 Claude rubric-based evaluation human evaluation symbolic subset