Found in the Middle: Calibrating Positional Attention Bias Improves Long
  Context Utilization
Large language models (LLMs), even when specifically trained to process long
input contexts, struggle to capture relevant information located in the middle
of their input. This phenomenon has been known as the lost-in-the-middle
problem. In this work, we make three contributions. First, we set out to
understand the factors that cause this phenomenon. In doing so, we establish a
connection between lost-in-the-middle to LLMs' intrinsic attention bias: LLMs
exhibit a U-shaped attention bias where the tokens at the beginning and at the
end of its input receive higher attention, regardless of their relevance.
Second, we mitigate this positional bias through a calibration mechanism,
found-in-the-middle, that allows the model to attend to contexts faithfully
according to their relevance, even though when they are in the middle. Third,
we show found-in-the-middle not only achieves better performance in locating
relevant information within a long context, but also eventually leads to
improved retrieval-augmented generation (RAG) performance across various tasks,
outperforming existing methods by up to 15 percentage points. These findings
open up future directions in understanding LLM attention bias and its potential
consequences.
667b2bc0c7f1674b87f472bf
Cheng-Yu Hsieh
claimed_verified
2024-07-11T08:27:20.500Z
6408e8663461c51cf73480e4
/avatars/7e4cbdb4b933181c673c9dc9ad041767.svg
Cheng-Yu Hsieh
user
cydhsieh01
667b2bc0c7f1674b87f472c0
Yung-Sung Chuang
claimed_verified
2024-10-31T09:55:37.130Z
5df84571da6d0311fd3d5407
https://cdn-avatars.huggingface.co/v1/production/uploads/1650651305661-5df84571da6d0311fd3d5407.png
Yung-Sung Chuang
user
voidism
667b2bc0c7f1674b87f472c1
Chun-Liang Li
667b2bc0c7f1674b87f472c2
Zifeng Wang
claimed_verified
2024-12-04T09:47:16.928Z
64da7b3bbb090cef5559fa31
/avatars/4eb059260f06837192c98927254cd89e.svg
Zifeng Wang
user
zifengw
667b2bc0c7f1674b87f472c3
Long T. Le
667b2bc0c7f1674b87f472c4
Abhishek Kumar
667b2bc0c7f1674b87f472c5
James Glass
667b2bc0c7f1674b87f472c6
Alexander Ratner
667b2bc0c7f1674b87f472c7
Chen-Yu Lee
extracted_confirmed
2024-06-25T21:03:17.172Z
64d1fe784dfd5df7076bfd1a
/avatars/e2bb108d1f6b1383f0e3f263c4747b3d.svg
Chen-Yu Lee
user
chenyulee
667b2bc0c7f1674b87f472c8
Ranjay Krishna
667b2bc0c7f1674b87f472c9
Tomas Pfister
Found in the Middle: Calibrating Positional Attention Bias Improves Long
  Context Utilization
Large language models (LLMs), even when specifically trained to process long
input contexts, struggle to capture relevant information located in the middle
of their input. This phenomenon has been known as the lost-in-the-middle
problem. In this work, we make three contributions. First, we set out to
understand the factors that cause this phenomenon. In doing so, we establish a
connection between lost-in-the-middle to LLMs' intrinsic attention bias: LLMs
exhibit a U-shaped attention bias where the tokens at the beginning and at the
end of its input receive higher attention, regardless of their relevance.
Second, we mitigate this positional bias through a calibration mechanism,
found-in-the-middle, that allows the model to attend to contexts faithfully
according to their relevance, even though when they are in the middle. Third,
we show found-in-the-middle not only achieves better performance in locating
relevant information within a long context, but also eventually leads to
improved retrieval-augmented generation (RAG) performance across various tasks,
outperforming existing methods by up to 15 percentage points. These findings
open up future directions in understanding LLM attention bias and its potential
consequences.
A calibration mechanism addresses the lost-in-the-middle problem in large language models, enhancing their ability to focus on relevant information and improving retrieval-augmented generation performance.
large language models lost-in-the-middle problem attention bias calibration mechanism retrieval-augmented generation