Attention Overflow: Language Model Input Blur during Long-Context
  Missing Items Recommendation
Large language models (LLMs) can suggest missing elements from items listed
in a prompt, which can be used for list completion or recommendations based on
users' history. However, their performance degrades when presented with too
many items, as they start to suggest items already included in the input list.
This occurs at around 100 items for mid-2024 flagship LLMs. We evaluate this
phenomenon on both synthetic problems (e.g., finding missing numbers in a given
range of shuffled integers) and realistic movie recommendation scenarios. We
refer to this issue as attention overflow, as preventing repetition
requires attending to all items simultaneously. Although iterative loops can
mitigate this problem, their costs increase with the repetition rate, affecting
the language models' ability to derive novelty from lengthy inputs.
669a125fac60504c1348eb8e
Damien Sileo
claimed_verified
2024-07-19T11:22:53.551Z
5fc0bcb41160c47d1d43856b
https://cdn-avatars.huggingface.co/v1/production/uploads/5fc0bcb41160c47d1d43856b/AHCEW4TfTdyjNBx-V_F5A.png
Damien Sileo
user
sileod
Attention Overflow: Language Model Input Blur during Long-Context
  Missing Items Recommendation
Large language models (LLMs) can suggest missing elements from items listed
in a prompt, which can be used for list completion or recommendations based on
users' history. However, their performance degrades when presented with too
many items, as they start to suggest items already included in the input list.
This occurs at around 100 items for mid-2024 flagship LLMs. We evaluate this
phenomenon on both synthetic problems (e.g., finding missing numbers in a given
range of shuffled integers) and realistic movie recommendation scenarios. We
refer to this issue as attention overflow, as preventing repetition
requires attending to all items simultaneously. Although iterative loops can
mitigate this problem, their costs increase with the repetition rate, affecting
the language models' ability to derive novelty from lengthy inputs.
Large language models struggle with suggesting missing elements from long lists due to attention overflow, affecting their performance and novelty derivation.
large language models list completion recommendations attention overflow