BABILong: Testing the Limits of LLMs with Long Context
  Reasoning-in-a-Haystack
In recent years, the input context sizes of large language models (LLMs) have
increased dramatically. However, existing evaluation methods have not kept
pace, failing to comprehensively assess the efficiency of models in handling
long contexts. To bridge this gap, we introduce the BABILong benchmark,
designed to test language models' ability to reason across facts distributed in
extremely long documents. BABILong includes a diverse set of 20 reasoning
tasks, including fact chaining, simple induction, deduction, counting, and
handling lists/sets. These tasks are challenging on their own, and even more
demanding when the required facts are scattered across long natural text. Our
evaluations show that popular LLMs effectively utilize only 10-20\% of the
context and their performance declines sharply with increased reasoning
complexity. Among alternatives to in-context reasoning, Retrieval-Augmented
Generation methods achieve a modest 60\% accuracy on single-fact question
answering, independent of context length. Among context extension methods, the
highest performance is demonstrated by recurrent memory transformers, enabling
the processing of lengths up to 11 million tokens. The BABILong benchmark is
extendable to any length to support the evaluation of new upcoming models with
increased capabilities, and we provide splits up to 1 million token lengths.
666ffb662b9e452739047aa9
Yuri Kuratov
claimed_verified
2024-06-17T13:43:02.813Z
618b9540682ec1c38327e586
https://cdn-avatars.huggingface.co/v1/production/uploads/618b9540682ec1c38327e586/v_ZBkfh8O9Zh6C2YQpuBX.jpeg
Yury Kuratov
user
yurakuratov
666ffb662b9e452739047aaa
Aydar Bulatov
admin_assigned
2024-06-17T14:13:46.699Z
64c8b321cb2f1bf0e7c0f54b
https://cdn-avatars.huggingface.co/v1/production/uploads/64c8b321cb2f1bf0e7c0f54b/JflXxMVnG9I0IB5YNyhXF.jpeg
Aydar Bulatov
user
booydar
666ffb662b9e452739047aab
Petr Anokhin
admin_assigned
2024-06-17T14:13:53.063Z
64f734b08e7fa6529fb693f1
/avatars/a50c7e1d22174f06fbaf99e8caca7c37.svg
Petr Anokhin
user
petranokhin
666ffb662b9e452739047aac
Ivan Rodkin
admin_assigned
2024-06-17T14:13:59.696Z
63c1ac8cc58fcfeac186bda2
https://cdn-avatars.huggingface.co/v1/production/uploads/63c1ac8cc58fcfeac186bda2/MajMhLgsvCBN81tmO9PZc.jpeg
Ivan Rodkin
user
irodkin
666ffb662b9e452739047aad
Dmitry Sorokin
666ffb662b9e452739047aae
Artyom Sorokin
admin_assigned
2024-06-17T14:14:09.088Z
64beb75986e7fb5b8a8fa731
/avatars/7053c64142300fa446ba5870eb28c3ee.svg
Artyom Sorokin
user
griver
666ffb662b9e452739047aaf
Mikhail Burtsev
admin_assigned
2024-06-17T14:14:16.679Z
639c6e978a34ed9a404c6a7b
/avatars/c98ca8c9f9ed8509c2f1bb6aa994fd57.svg
MIKHAIL BURTSEV
user
mbur
BABILong: Testing the Limits of LLMs with Long Context
  Reasoning-in-a-Haystack
In recent years, the input context sizes of large language models (LLMs) have
increased dramatically. However, existing evaluation methods have not kept
pace, failing to comprehensively assess the efficiency of models in handling
long contexts. To bridge this gap, we introduce the BABILong benchmark,
designed to test language models' ability to reason across facts distributed in
extremely long documents. BABILong includes a diverse set of 20 reasoning
tasks, including fact chaining, simple induction, deduction, counting, and
handling lists/sets. These tasks are challenging on their own, and even more
demanding when the required facts are scattered across long natural text. Our
evaluations show that popular LLMs effectively utilize only 10-20\% of the
context and their performance declines sharply with increased reasoning
complexity. Among alternatives to in-context reasoning, Retrieval-Augmented
Generation methods achieve a modest 60\% accuracy on single-fact question
answering, independent of context length. Among context extension methods, the
highest performance is demonstrated by recurrent memory transformers, enabling
the processing of lengths up to 11 million tokens. The BABILong benchmark is
extendable to any length to support the evaluation of new upcoming models with
increased capabilities, and we provide splits up to 1 million token lengths.
BABILong benchmark evaluates large language models' reasoning across long documents, revealing that models use only a small portion of context and that recurrent memory transformers handle longer contexts better.
large language models BABILong benchmark fact chaining simple induction deduction counting handling lists/sets Retrieval-Augmented Generation recurrent memory transformers