ReMamba: Equip Mamba with Effective Long-Sequence Modeling
While the Mamba architecture demonstrates superior inference efficiency and
competitive performance on short-context natural language processing (NLP)
tasks, empirical evidence suggests its capacity to comprehend long contexts is
limited compared to transformer-based models. In this study, we investigate the
long-context efficiency issues of the Mamba models and propose ReMamba, which
enhances Mamba's ability to comprehend long contexts. ReMamba incorporates
selective compression and adaptation techniques within a two-stage re-forward
process, incurring minimal additional inference costs overhead. Experimental
results on the LongBench and L-Eval benchmarks demonstrate ReMamba's efficacy,
improving over the baselines by 3.2 and 1.6 points, respectively, and attaining
performance almost on par with same-size transformer models.
66cfe43588a57d59e6cabe59
Danlong Yuan
admin_assigned
2024-08-29T08:00:54.734Z
6567ea516b1e4d61d8e16e46
/avatars/ff33037da2a5f35377e1854e77b5f1ff.svg
Yuan Danlong
user
lblankl
66cfe43588a57d59e6cabe5a
Jiahao Liu
claimed_verified
2024-08-29T08:57:42.071Z
64b75e6e9ebb7e6c7dd5f7c5
/avatars/5173c10cce77c429802e51e3a28368df.svg
glaicer liu
user
glacier000
66cfe43588a57d59e6cabe5b
Bei Li
66cfe43588a57d59e6cabe5c
Huishuai Zhang
66cfe43588a57d59e6cabe5d
Jingang Wang
admin_assigned
2024-08-29T08:01:34.650Z
647097cbcfd57849518e656b
/avatars/c66fe0add29c1bde9e3a98bf4a8793b9.svg
Jingang Wang
user
bitwjg
66cfe43588a57d59e6cabe5e
Xunliang Cai
66cfe43588a57d59e6cabe5f
Dongyan Zhao
ReMamba: Equip Mamba with Effective Long-Sequence Modeling
While the Mamba architecture demonstrates superior inference efficiency and
competitive performance on short-context natural language processing (NLP)
tasks, empirical evidence suggests its capacity to comprehend long contexts is
limited compared to transformer-based models. In this study, we investigate the
long-context efficiency issues of the Mamba models and propose ReMamba, which
enhances Mamba's ability to comprehend long contexts. ReMamba incorporates
selective compression and adaptation techniques within a two-stage re-forward
process, incurring minimal additional inference costs overhead. Experimental
results on the LongBench and L-Eval benchmarks demonstrate ReMamba's efficacy,
improving over the baselines by 3.2 and 1.6 points, respectively, and attaining
performance almost on par with same-size transformer models.
ReMamba addresses Mamba's long-context efficiency limitations through selective compression and adaptation, narrowing the performance gap with transformer models on long contexts.
inference efficiency long-context natural language processing transformer-based models selective compression adaptation techniques two-stage re-forward process LongBench L-Eval performance gaps