Needle Threading: Can LLMs Follow Threads through Near-Million-Scale
  Haystacks?
As the context limits of Large Language Models (LLMs) increase, the range of
possible applications and downstream functions broadens. In many real-world
tasks, decisions depend on details scattered across collections of often
disparate documents containing mostly irrelevant information. Long-context LLMs
appear well-suited to this form of complex information retrieval and reasoning,
which has traditionally proven costly and time-consuming. However, although the
development of longer context models has seen rapid gains in recent years, our
understanding of how effectively LLMs use their context has not kept pace. To
address this, we conduct a set of retrieval experiments designed to evaluate
the capabilities of 17 leading LLMs, such as their ability to follow threads of
information through the context window. Strikingly, we find that many models
are remarkably threadsafe: capable of simultaneously following multiple threads
without significant loss in performance. Still, for many models, we find the
effective context limit is significantly shorter than the supported context
length, with accuracy decreasing as the context window grows. Our study also
highlights the important point that token counts from different tokenizers
should not be directly compared -- they often correspond to substantially
different numbers of written characters. We release our code and long-context
experimental data.
672deabffbdaf22d9db944c0
Jonathan Roberts
claimed_verified
2024-11-15T09:29:08.971Z
632456d21ed511c0c5231afd
https://cdn-avatars.huggingface.co/v1/production/uploads/1671535517748-632456d21ed511c0c5231afd.jpeg
Jonathan Roberts
user
jonathan-roberts1
672deabffbdaf22d9db944c1
Kai Han
672deabffbdaf22d9db944c2
Samuel Albanie
Needle Threading: Can LLMs Follow Threads through Near-Million-Scale
  Haystacks?
As the context limits of Large Language Models (LLMs) increase, the range of
possible applications and downstream functions broadens. In many real-world
tasks, decisions depend on details scattered across collections of often
disparate documents containing mostly irrelevant information. Long-context LLMs
appear well-suited to this form of complex information retrieval and reasoning,
which has traditionally proven costly and time-consuming. However, although the
development of longer context models has seen rapid gains in recent years, our
understanding of how effectively LLMs use their context has not kept pace. To
address this, we conduct a set of retrieval experiments designed to evaluate
the capabilities of 17 leading LLMs, such as their ability to follow threads of
information through the context window. Strikingly, we find that many models
are remarkably threadsafe: capable of simultaneously following multiple threads
without significant loss in performance. Still, for many models, we find the
effective context limit is significantly shorter than the supported context
length, with accuracy decreasing as the context window grows. Our study also
highlights the important point that token counts from different tokenizers
should not be directly compared -- they often correspond to substantially
different numbers of written characters. We release our code and long-context
experimental data.
Research evaluates the information retrieval and reasoning capabilities of 17 long-context LLMs, revealing that many can handle multiple threads of information efficiently within a limited effective context window.
LLMs long-context models context window information retrieval reasoning threadsafe tokenizers