Eliminating Position Bias of Language Models: A Mechanistic Approach
Position bias has proven to be a prevalent issue of modern language models
(LMs), where the models prioritize content based on its position within the
given context. This bias often leads to unexpected model failures and hurts
performance, robustness, and reliability across various applications. Our
mechanistic analysis attributes the position bias to two components employed in
nearly all state-of-the-art LMs: causal attention and relative positional
encodings. Specifically, we find that causal attention generally causes models
to favor distant content, while relative positional encodings like RoPE prefer
nearby ones based on the analysis of retrieval-augmented question answering
(QA). Further, our empirical study on object detection reveals that position
bias is also present in vision-language models (VLMs).
  Based on the above analyses, we propose to ELIMINATE position bias caused by
different input segment orders (e.g., options in LM-as-a-judge, retrieved
documents in QA) in a TRAINING-FREE ZERO-SHOT manner. Our method changes the
causal attention to bidirectional attention between segments and utilizes model
attention values to decide the relative orders of segments instead of using the
order provided in input prompts, therefore enabling Position-INvariant
inferencE (PINE) at the segment level. By eliminating position bias, models
achieve better performance and reliability in downstream tasks where position
bias widely exists, such as LM-as-a-judge and retrieval-augmented QA.
  Notably, PINE is especially useful when adapting LMs for evaluating reasoning
pairs: it consistently provides 8 to 10 percentage points performance gains in
most cases, and makes Llama-3-70B-Instruct perform even better than
GPT-4-0125-preview on the RewardBench reasoning subset.
668365c0f8fd8217cc21a8a3
Ziqi Wang
claimed_verified
2024-07-05T06:55:41.246Z
651c321c4e15c57e1b0efca6
/avatars/241c4d975356bd986bad7303e1534bce.svg
Ziqi wang
user
wzq016
668365c0f8fd8217cc21a8a4
Hanlin Zhang
claimed_verified
2024-12-11T09:23:26.518Z
624054bcc2c17da6a63eb539
/avatars/bf52dc0683b4100733f8696a97696d0e.svg
hlzhang109
user
hlzhang109
668365c0f8fd8217cc21a8a5
Xiner Li
668365c0f8fd8217cc21a8a6
Kuan-Hao Huang
claimed_verified
2025-07-28T08:48:20.585Z
658a57b4126b8d7eae07b983
https://cdn-avatars.huggingface.co/v1/production/uploads/658a57b4126b8d7eae07b983/reZrbaObwq_kNVsuTR_iR.jpeg
Kuan-Hao Huang
user
kuanhaoh
668365c0f8fd8217cc21a8a7
Chi Han
claimed_verified
2025-03-18T08:12:55.939Z
645b11ff5987734177a7cdcf
/avatars/3b04d2ca166eb28f227fc8d800e55c1c.svg
Chi Han
user
Glaciohound
668365c0f8fd8217cc21a8a8
Shuiwang Ji
668365c0f8fd8217cc21a8a9
Sham M. Kakade
668365c0f8fd8217cc21a8aa
Hao Peng
668365c0f8fd8217cc21a8ab
Heng Ji
Eliminating Position Bias of Language Models: A Mechanistic Approach
Position bias has proven to be a prevalent issue of modern language models
(LMs), where the models prioritize content based on its position within the
given context. This bias often leads to unexpected model failures and hurts
performance, robustness, and reliability across various applications. Our
mechanistic analysis attributes the position bias to two components employed in
nearly all state-of-the-art LMs: causal attention and relative positional
encodings. Specifically, we find that causal attention generally causes models
to favor distant content, while relative positional encodings like RoPE prefer
nearby ones based on the analysis of retrieval-augmented question answering
(QA). Further, our empirical study on object detection reveals that position
bias is also present in vision-language models (VLMs).
  Based on the above analyses, we propose to ELIMINATE position bias caused by
different input segment orders (e.g., options in LM-as-a-judge, retrieved
documents in QA) in a TRAINING-FREE ZERO-SHOT manner. Our method changes the
causal attention to bidirectional attention between segments and utilizes model
attention values to decide the relative orders of segments instead of using the
order provided in input prompts, therefore enabling Position-INvariant
inferencE (PINE) at the segment level. By eliminating position bias, models
achieve better performance and reliability in downstream tasks where position
bias widely exists, such as LM-as-a-judge and retrieval-augmented QA.
  Notably, PINE is especially useful when adapting LMs for evaluating reasoning
pairs: it consistently provides 8 to 10 percentage points performance gains in
most cases, and makes Llama-3-70B-Instruct perform even better than
GPT-4-0125-preview on the RewardBench reasoning subset.
The proposed Position-Invariant Inference (PINE) method eliminates position bias in language models by changing causal to bidirectional attention, improving performance and reliability in tasks like LM-as-a-judge and retrieval-augmented QA.
positional bias causal attention relative positional encodings RoPE feature detection object detection vision-language models VLMs bidirectional attention RewardBench reasoning subset