Attention Basin: Why Contextual Position Matters in Large Language
  Models
The performance of Large Language Models (LLMs) is significantly sensitive to
the contextual position of information in the input. To investigate the
mechanism behind this positional bias, our extensive experiments reveal a
consistent phenomenon we term the attention basin: when presented with a
sequence of structured items (e.g., retrieved documents or few-shot examples),
models systematically assign higher attention to the items at the beginning and
end of the sequence, while neglecting those in the middle. Crucially, our
analysis further reveals that allocating higher attention to critical
information is key to enhancing model performance. Based on these insights, we
introduce Attention-Driven Reranking (AttnRank), a two-stage framework that (i)
estimates a model's intrinsic positional attention preferences using a small
calibration set, and (ii) reorders retrieved documents or few-shot examples to
align the most salient content with these high-attention positions. AttnRank is
a model-agnostic, training-free, and plug-and-play method with minimal
computational overhead. Experiments on multi-hop QA and few-shot in-context
learning tasks demonstrate that AttnRank achieves substantial improvements
across 10 large language models of varying architectures and scales, without
modifying model parameters or training procedures.
689625d8e78b624607aa96fc
Zihao Yi
claimed_verified
2025-08-11T06:50:26.682Z
65f7ad5526f86cf3378a59f2
/avatars/752aa3c072c71ece41ea786371574777.svg
Zihao Yi
user
Zihao1
689625d8e78b624607aa96fd
Delong Zeng
689625d8e78b624607aa96fe
Zhenqing Ling
689625d8e78b624607aa96ff
Haohao Luo
claimed_verified
2025-08-14T13:39:58.079Z
64c0dc78e56520a63d404c9b
https://cdn-avatars.huggingface.co/v1/production/uploads/64c0dc78e56520a63d404c9b/kg8hXfR2DU4kfOkQ4OXvw.jpeg
Luohh_SYSU
user
Lhh123
689625d8e78b624607aa9700
Zhe Xu
689625d8e78b624607aa9701
Wei Liu
689625d8e78b624607aa9702
Jian Luan
689625d8e78b624607aa9703
Wanxia Cao
689625d8e78b624607aa9704
Ying Shen
Attention Basin: Why Contextual Position Matters in Large Language
  Models
The performance of Large Language Models (LLMs) is significantly sensitive to
the contextual position of information in the input. To investigate the
mechanism behind this positional bias, our extensive experiments reveal a
consistent phenomenon we term the attention basin: when presented with a
sequence of structured items (e.g., retrieved documents or few-shot examples),
models systematically assign higher attention to the items at the beginning and
end of the sequence, while neglecting those in the middle. Crucially, our
analysis further reveals that allocating higher attention to critical
information is key to enhancing model performance. Based on these insights, we
introduce Attention-Driven Reranking (AttnRank), a two-stage framework that (i)
estimates a model's intrinsic positional attention preferences using a small
calibration set, and (ii) reorders retrieved documents or few-shot examples to
align the most salient content with these high-attention positions. AttnRank is
a model-agnostic, training-free, and plug-and-play method with minimal
computational overhead. Experiments on multi-hop QA and few-shot in-context
learning tasks demonstrate that AttnRank achieves substantial improvements
across 10 large language models of varying architectures and scales, without
modifying model parameters or training procedures.