Evaluating Language Model Context Windows: A "Working Memory" Test and
  Inference-time Correction
Large language models are prominently used in real-world applications, often
tasked with reasoning over large volumes of documents. An exciting development
in this space is models boasting extended context capabilities, with some
accommodating over 2 million tokens. Such long context model capabilities
remain uncertain in production systems, motivating the need to benchmark their
performance on real world use cases. We address this challenge by proposing
SWiM, an evaluation framework that addresses the limitations of standard tests.
Testing the framework on eight long context models, we find that even strong
models such as GPT-4 and Claude 3 Opus degrade in performance when information
is present in the middle of the context window (lost-in-the-middle effect).
Next, in addition to our benchmark, we propose medoid voting, a simple, but
effective training-free approach that helps alleviate this effect, by
generating responses a few times, each time randomly permuting documents in the
context, and selecting the medoid answer. We evaluate medoid voting on single
document QA tasks, achieving up to a 24% lift in accuracy.
668d67ba63d1bc005bd5e328
Amanda Dsouza
claimed_verified
2024-07-09T21:09:03.551Z
654d53ac6db202b835339be9
/avatars/01d3046e63b3867d5faa6d64cdef09d7.svg
Amanda Dsouza
user
andsouzasnorkelai
668d67ba63d1bc005bd5e329
Christopher Glaze
claimed_verified
2024-07-09T21:09:01.496Z
644985b8eb7db8f70fb72fab
/avatars/f9861800bd9ad0645af2da9d0db4bcbc.svg
Christopher Glaze
user
cmglaze
668d67ba63d1bc005bd5e32a
Changho Shin
668d67ba63d1bc005bd5e32b
Frederic Sala
claimed_verified
2024-07-10T08:19:58.464Z
63044153fc783bfc74461fa9
https://cdn-avatars.huggingface.co/v1/production/uploads/63044153fc783bfc74461fa9/O5Jz6VPvfVHFyaXH_XqoN.jpeg
Frederic Sala
user
fredsala
Evaluating Language Model Context Windows: A "Working Memory" Test and
  Inference-time Correction
Large language models are prominently used in real-world applications, often
tasked with reasoning over large volumes of documents. An exciting development
in this space is models boasting extended context capabilities, with some
accommodating over 2 million tokens. Such long context model capabilities
remain uncertain in production systems, motivating the need to benchmark their
performance on real world use cases. We address this challenge by proposing
SWiM, an evaluation framework that addresses the limitations of standard tests.
Testing the framework on eight long context models, we find that even strong
models such as GPT-4 and Claude 3 Opus degrade in performance when information
is present in the middle of the context window (lost-in-the-middle effect).
Next, in addition to our benchmark, we propose medoid voting, a simple, but
effective training-free approach that helps alleviate this effect, by
generating responses a few times, each time randomly permuting documents in the
context, and selecting the medoid answer. We evaluate medoid voting on single
document QA tasks, achieving up to a 24% lift in accuracy.
SWiM evaluates long context models and proposes medoid voting to mitigate the lost-in-the-middle effect, significantly improving accuracy in single document QA tasks.
large language models extended context capabilities SWiM benchmark long context models lost-in-the-middle effect medoid voting single document QA