Differential Mamba
Sequence models like Transformers and RNNs often overallocate attention to
irrelevant context, leading to noisy intermediate representations. This
degrades LLM capabilities by promoting hallucinations, weakening long-range and
retrieval abilities, and reducing robustness. Recent work has shown that
differential design can mitigate this issue in Transformers, improving their
effectiveness across various applications. In this paper, we explore whether
these techniques, originally developed for Transformers, can be applied to
Mamba, a recent architecture based on selective state-space layers that
achieves Transformer-level performance with greater efficiency. We show that a
naive adaptation of differential design to Mamba is insufficient and requires
careful architectural modifications. To address this, we introduce a novel
differential mechanism for Mamba, empirically validated on language modeling
benchmarks, demonstrating improved retrieval capabilities and superior
performance over vanilla Mamba. Finally, we conduct extensive ablation studies
and empirical analyses to justify our design choices and provide evidence that
our approach effectively mitigates the overallocation problem in Mamba-based
models. Our code is publicly available.
686e0c68cb5725779c60b4ed
Nadav Schneider
claimed_verified
2025-07-13T07:26:33.505Z
65ce0b4a03a8179f5da5d4ef
/avatars/c0b68efb885486c5180d1d69c4e317ac.svg
Nadav Schneider
user
nadavsc
686e0c68cb5725779c60b4ee
Itamar Zimerman
686e0c68cb5725779c60b4ef
Eliya Nachmani
Differential Mamba
Sequence models like Transformers and RNNs often overallocate attention to
irrelevant context, leading to noisy intermediate representations. This
degrades LLM capabilities by promoting hallucinations, weakening long-range and
retrieval abilities, and reducing robustness. Recent work has shown that
differential design can mitigate this issue in Transformers, improving their
effectiveness across various applications. In this paper, we explore whether
these techniques, originally developed for Transformers, can be applied to
Mamba, a recent architecture based on selective state-space layers that
achieves Transformer-level performance with greater efficiency. We show that a
naive adaptation of differential design to Mamba is insufficient and requires
careful architectural modifications. To address this, we introduce a novel
differential mechanism for Mamba, empirically validated on language modeling
benchmarks, demonstrating improved retrieval capabilities and superior
performance over vanilla Mamba. Finally, we conduct extensive ablation studies
and empirical analyses to justify our design choices and provide evidence that
our approach effectively mitigates the overallocation problem in Mamba-based
models. Our code is publicly available.
A novel differential mechanism for Mamba, a selective state-space layer architecture, improves language modeling performance by addressing overallocation of attention to irrelevant context.
Transformers RNNs differential design Mamba selective state-space layers language modeling retrieval capabilities ablation studies empirical analyses