Can Mamba Learn How to Learn? A Comparative Study on In-Context Learning
  Tasks
State-space models (SSMs), such as Mamba Gu & Dao (2034), have been proposed
as alternatives to Transformer networks in language modeling, by incorporating
gating, convolutions, and input-dependent token selection to mitigate the
quadratic cost of multi-head attention. Although SSMs exhibit competitive
performance, their in-context learning (ICL) capabilities, a remarkable
emergent property of modern language models that enables task execution without
parameter optimization, remain underexplored compared to Transformers. In this
study, we evaluate the ICL performance of SSMs, focusing on Mamba, against
Transformer models across various tasks. Our results show that SSMs perform
comparably to Transformers in standard regression ICL tasks, while
outperforming them in tasks like sparse parity learning. However, SSMs fall
short in tasks involving non-standard retrieval functionality. To address these
limitations, we introduce a hybrid model, \variant, that combines Mamba with
attention blocks, surpassing individual models in tasks where they struggle
independently. Our findings suggest that hybrid architectures offer promising
avenues for enhancing ICL in language models.
65c2f90a82e4546db95c26d8
Jongho Park
65c2f90a82e4546db95c26d9
Jaeseung Park
65c2f90a82e4546db95c26da
Zheyang Xiong
claimed_verified
2024-10-11T07:46:44.350Z
67084dbb652deb3c09d8f633
/avatars/6489cc414ec6b7be3523e335ba14888f.svg
Zheyang Xiong
user
zheyang-hf
65c2f90a82e4546db95c26db
Nayoung Lee
65c2f90a82e4546db95c26dc
Jaewoong Cho
65c2f90a82e4546db95c26dd
Samet Oymak
65c2f90a82e4546db95c26de
Kangwook Lee
65c2f90a82e4546db95c26df
Dimitris Papailiopoulos
extracted_pending
2024-02-07T03:29:15.708Z
64428cf07feb866811b0be6a
/avatars/f30d339a02ce070c7e77315b59e20359.svg
Dimitris Papailiopoulos
user
anadim
Can Mamba Learn How to Learn? A Comparative Study on In-Context Learning
  Tasks
State-space models (SSMs), such as Mamba Gu & Dao (2034), have been proposed
as alternatives to Transformer networks in language modeling, by incorporating
gating, convolutions, and input-dependent token selection to mitigate the
quadratic cost of multi-head attention. Although SSMs exhibit competitive
performance, their in-context learning (ICL) capabilities, a remarkable
emergent property of modern language models that enables task execution without
parameter optimization, remain underexplored compared to Transformers. In this
study, we evaluate the ICL performance of SSMs, focusing on Mamba, against
Transformer models across various tasks. Our results show that SSMs perform
comparably to Transformers in standard regression ICL tasks, while
outperforming them in tasks like sparse parity learning. However, SSMs fall
short in tasks involving non-standard retrieval functionality. To address these
limitations, we introduce a hybrid model, \variant, that combines Mamba with
attention blocks, surpassing individual models in tasks where they struggle
independently. Our findings suggest that hybrid architectures offer promising
avenues for enhancing ICL in language models.
SSMs, including Mamba, offer competitive ICL capabilities in standard regression tasks but lag in non-standard retrieval; a hybrid model combining Mamba with attention blocks enhances ICL performance.
state-space models SSMs Mamba Transformer networks language modeling gating convolutions input-dependent token selection multi-head attention in-context learning ICL standard regression sparse parity learning non-standard retrieval attention blocks hybrid model