Masking in Multi-hop QA: An Analysis of How Language Models Perform with
  Context Permutation
Multi-hop Question Answering (MHQA) adds layers of complexity to question
answering, making it more challenging. When Language Models (LMs) are prompted
with multiple search results, they are tasked not only with retrieving relevant
information but also employing multi-hop reasoning across the information
sources. Although LMs perform well on traditional question-answering tasks, the
causal mask can hinder their capacity to reason across complex contexts. In
this paper, we explore how LMs respond to multi-hop questions by permuting
search results (retrieved documents) under various configurations. Our study
reveals interesting findings as follows: 1) Encoder-decoder models, such as the
ones in the Flan-T5 family, generally outperform causal decoder-only LMs in
MHQA tasks, despite being significantly smaller in size; 2) altering the order
of gold documents reveals distinct trends in both Flan T5 models and fine-tuned
decoder-only models, with optimal performance observed when the document order
aligns with the reasoning chain order; 3) enhancing causal decoder-only models
with bi-directional attention by modifying the causal mask can effectively
boost their end performance. In addition to the above, we conduct a thorough
investigation of the distribution of LM attention weights in the context of
MHQA. Our experiments reveal that attention weights tend to peak at higher
values when the resulting answer is correct. We leverage this finding to
heuristically improve LMs' performance on this task. Our code is publicly
available at https://github.com/hwy9855/MultiHopQA-Reasoning.
682bdc203fff9e115285b2b6
Wenyu Huang
claimed_verified
2025-05-20T07:21:08.965Z
64c4d03ca684146b1cdb8237
/avatars/82b513c686ab20c859ca900fd59dfac7.svg
Wenyu Huang
user
hwy9855
682bdc203fff9e115285b2b7
Pavlos Vougiouklis
extracted_confirmed
2025-05-21T11:43:19.081Z
641c83a492cd2530299714c7
/avatars/928195163f9040a2649953c86b9d6c79.svg
Pavlos V
user
pvougiou
682bdc203fff9e115285b2b8
Mirella Lapata
682bdc203fff9e115285b2b9
Jeff Z. Pan
Masking in Multi-hop QA: An Analysis of How Language Models Perform with
  Context Permutation
Multi-hop Question Answering (MHQA) adds layers of complexity to question
answering, making it more challenging. When Language Models (LMs) are prompted
with multiple search results, they are tasked not only with retrieving relevant
information but also employing multi-hop reasoning across the information
sources. Although LMs perform well on traditional question-answering tasks, the
causal mask can hinder their capacity to reason across complex contexts. In
this paper, we explore how LMs respond to multi-hop questions by permuting
search results (retrieved documents) under various configurations. Our study
reveals interesting findings as follows: 1) Encoder-decoder models, such as the
ones in the Flan-T5 family, generally outperform causal decoder-only LMs in
MHQA tasks, despite being significantly smaller in size; 2) altering the order
of gold documents reveals distinct trends in both Flan T5 models and fine-tuned
decoder-only models, with optimal performance observed when the document order
aligns with the reasoning chain order; 3) enhancing causal decoder-only models
with bi-directional attention by modifying the causal mask can effectively
boost their end performance. In addition to the above, we conduct a thorough
investigation of the distribution of LM attention weights in the context of
MHQA. Our experiments reveal that attention weights tend to peak at higher
values when the resulting answer is correct. We leverage this finding to
heuristically improve LMs' performance on this task. Our code is publicly
available at https://github.com/hwy9855/MultiHopQA-Reasoning.
Encoder-decoder models outperform causal decoder-only models in multi-hop question answering by leveraging permutation of search results and bi-directional attention.
multi-hop question answering encoder-decoder models causal decoder-only models Flan-T5 multi-hop reasoning causal mask bi-directional attention attention weights