BIG-Bench Extra Hard
Large language models (LLMs) are increasingly deployed in everyday
applications, demanding robust general reasoning capabilities and diverse
reasoning skillset. However, current LLM reasoning benchmarks predominantly
focus on mathematical and coding abilities, leaving a gap in evaluating broader
reasoning proficiencies. One particular exception is the BIG-Bench dataset,
which has served as a crucial benchmark for evaluating the general reasoning
capabilities of LLMs, thanks to its diverse set of challenging tasks that
allowed for a comprehensive assessment of general reasoning across various
skills within a unified framework. However, recent advances in LLMs have led to
saturation on BIG-Bench, and its harder version BIG-Bench Hard (BBH).
State-of-the-art models achieve near-perfect scores on many tasks in BBH, thus
diminishing its utility. To address this limitation, we introduce BIG-Bench
Extra Hard (BBEH), a new benchmark designed to push the boundaries of LLM
reasoning evaluation. BBEH replaces each task in BBH with a novel task that
probes a similar reasoning capability but exhibits significantly increased
difficulty. We evaluate various models on BBEH and observe a (harmonic) average
accuracy of 9.8\% for the best general-purpose model and 44.8\% for the best
reasoning-specialized model, indicating substantial room for improvement and
highlighting the ongoing challenge of achieving robust general reasoning in
LLMs. We release BBEH publicly at: https://github.com/google-deepmind/bbeh.
67c01747e8c7d56a8e0cbdc3
Mehran Kazemi
67c01747e8c7d56a8e0cbdc4
Bahare Fatemi
extracted_pending
2025-02-27T07:42:00.525Z
654e97ef5da3196a78409341
/avatars/1a5ea7351ca21960891cf9721b9f4667.svg
Bahare Fatemi
user
baharefatemi
67c01747e8c7d56a8e0cbdc5
Hritik Bansal
67c01747e8c7d56a8e0cbdc6
John Palowitch
67c01747e8c7d56a8e0cbdc7
Chrysovalantis Anastasiou
67c01747e8c7d56a8e0cbdc8
Sanket Vaibhav Mehta
67c01747e8c7d56a8e0cbdc9
Lalit K. Jain
67c01747e8c7d56a8e0cbdca
Virginia Aglietti
67c01747e8c7d56a8e0cbdcb
Disha Jindal
67c01747e8c7d56a8e0cbdcc
Peter Chen
67c01747e8c7d56a8e0cbdcd
Nishanth Dikkala
67c01747e8c7d56a8e0cbdce
Gladys Tyen
67c01747e8c7d56a8e0cbdcf
Xin Liu
67c01747e8c7d56a8e0cbdd0
Uri Shalit
67c01747e8c7d56a8e0cbdd1
Silvia Chiappa
67c01747e8c7d56a8e0cbdd2
Kate Olszewska
67c01747e8c7d56a8e0cbdd3
Yi Tay
67c01747e8c7d56a8e0cbdd4
Vinh Q. Tran
67c01747e8c7d56a8e0cbdd5
Quoc V. Le
67c01747e8c7d56a8e0cbdd6
Orhan Firat
BIG-Bench Extra Hard
Large language models (LLMs) are increasingly deployed in everyday
applications, demanding robust general reasoning capabilities and diverse
reasoning skillset. However, current LLM reasoning benchmarks predominantly
focus on mathematical and coding abilities, leaving a gap in evaluating broader
reasoning proficiencies. One particular exception is the BIG-Bench dataset,
which has served as a crucial benchmark for evaluating the general reasoning
capabilities of LLMs, thanks to its diverse set of challenging tasks that
allowed for a comprehensive assessment of general reasoning across various
skills within a unified framework. However, recent advances in LLMs have led to
saturation on BIG-Bench, and its harder version BIG-Bench Hard (BBH).
State-of-the-art models achieve near-perfect scores on many tasks in BBH, thus
diminishing its utility. To address this limitation, we introduce BIG-Bench
Extra Hard (BBEH), a new benchmark designed to push the boundaries of LLM
reasoning evaluation. BBEH replaces each task in BBH with a novel task that
probes a similar reasoning capability but exhibits significantly increased
difficulty. We evaluate various models on BBEH and observe a (harmonic) average
accuracy of 9.8\% for the best general-purpose model and 44.8\% for the best
reasoning-specialized model, indicating substantial room for improvement and
highlighting the ongoing challenge of achieving robust general reasoning in
LLMs. We release BBEH publicly at: https://github.com/google-deepmind/bbeh.
BIG-Bench Extra Hard (BBEH) introduces new tasks to challenge LLMs, showing significant room for improvement in general reasoning capabilities beyond current benchmarks.
BIG-Bench BIG-Bench Hard (BBH) BIG-Bench Extra Hard (BBEH) large language models (LLMs) reasoning benchmarks general reasoning capabilities reasoning skillset mathematical abilities coding abilities diverse set of challenging tasks comprehensive assessment general reasoning unified framework recent advances saturation harmonic average accuracy general-purpose model reasoning-specialized model