Lost in the Middle: How Language Models Use Long Contexts
While recent language models have the ability to take long contexts as input,
relatively little is known about how well the language models use longer
context. We analyze language model performance on two tasks that require
identifying relevant information within their input contexts: multi-document
question answering and key-value retrieval. We find that performance is often
highest when relevant information occurs at the beginning or end of the input
context, and significantly degrades when models must access relevant
information in the middle of long contexts. Furthermore, performance
substantially decreases as the input context grows longer, even for explicitly
long-context models. Our analysis provides a better understanding of how
language models use their input context and provides new evaluation protocols
for future long-context models.
64a784267fc46ddd14e16f2d
Nelson F. Liu
64a784267fc46ddd14e16f2e
Kevin Lin
claimed_verified
2023-07-10T09:24:26.156Z
64a85bfad14398e270836ef8
/avatars/61e4d298216cb89c0ec070e95d229ab6.svg
Kevin Lin
user
nlpkevinl
64a784267fc46ddd14e16f2f
John Hewitt
64a784267fc46ddd14e16f30
Ashwin Paranjape
64a784267fc46ddd14e16f31
Michele Bevilacqua
64a784267fc46ddd14e16f32
Fabio Petroni
64a784267fc46ddd14e16f33
Percy Liang
Lost in the Middle: How Language Models Use Long Contexts
While recent language models have the ability to take long contexts as input,
relatively little is known about how well the language models use longer
context. We analyze language model performance on two tasks that require
identifying relevant information within their input contexts: multi-document
question answering and key-value retrieval. We find that performance is often
highest when relevant information occurs at the beginning or end of the input
context, and significantly degrades when models must access relevant
information in the middle of long contexts. Furthermore, performance
substantially decreases as the input context grows longer, even for explicitly
long-context models. Our analysis provides a better understanding of how
language models use their input context and provides new evaluation protocols
for future long-context models.
Analysis reveals language models perform better when relevant information is at the ends of longer contexts and suggests new evaluation protocols for long-context models.
multi-document question answering key-value retrieval long-context models