Hyper-multi-step: The Truth Behind Difficult Long-context Tasks
Long-context language models (LCLM), characterized by their extensive context
window, is becoming increasingly popular. Meanwhile, many long-context
benchmarks present challenging tasks that even the most advanced LCLMs struggle
to complete. However, the underlying sources of various challenging
long-context tasks have seldom been studied. To bridge this gap, we conduct
experiments to indicate their difficulty stems primarily from two basic issues:
"multi-matching retrieval," which requires the simultaneous retrieval of
multiple items, and "logic-based retrieval," which necessitates logical
judgment within retrieval criteria. These two problems, while seemingly
straightforward, actually exceed the capabilities of LCLMs because they are
proven to be hyper-multi-step (demanding numerous steps to solve) in nature.
This finding could explain why LLMs struggle with more advanced long-context
tasks, providing a more accurate perspective for rethinking solutions for them.
6706260a18b86c7e905baa92
Yijiong Yu
claimed_verified
2024-10-09T07:37:08.228Z
6374c494958cd71fa7ea0a9d
https://cdn-avatars.huggingface.co/v1/production/uploads/6374c494958cd71fa7ea0a9d/2YCKv6tXCZXtsIOFIIXjs.png
yuyijiong
user
yuyijiong
Hyper-multi-step: The Truth Behind Difficult Long-context Tasks
Long-context language models (LCLM), characterized by their extensive context
window, is becoming increasingly popular. Meanwhile, many long-context
benchmarks present challenging tasks that even the most advanced LCLMs struggle
to complete. However, the underlying sources of various challenging
long-context tasks have seldom been studied. To bridge this gap, we conduct
experiments to indicate their difficulty stems primarily from two basic issues:
"multi-matching retrieval," which requires the simultaneous retrieval of
multiple items, and "logic-based retrieval," which necessitates logical
judgment within retrieval criteria. These two problems, while seemingly
straightforward, actually exceed the capabilities of LCLMs because they are
proven to be hyper-multi-step (demanding numerous steps to solve) in nature.
This finding could explain why LLMs struggle with more advanced long-context
tasks, providing a more accurate perspective for rethinking solutions for them.
Detailed experiments reveal that long-context language models struggle with tasks primarily due to hyper-multi-step challenges like multi-matching and logic-based retrieval.
long-context language models multi-matching retrieval logic-based retrieval hyper-multi-step