DenseMamba: State Space Models with Dense Hidden Connection for
  Efficient Large Language Models
Large language models (LLMs) face a daunting challenge due to the excessive
computational and memory requirements of the commonly used Transformer
architecture. While state space model (SSM) is a new type of foundational
network architecture offering lower computational complexity, their performance
has yet to fully rival that of Transformers. This paper introduces DenseSSM, a
novel approach to enhance the flow of hidden information between layers in
SSMs. By selectively integrating shallowlayer hidden states into deeper layers,
DenseSSM retains fine-grained information crucial for the final output. Dense
connections enhanced DenseSSM still maintains the training parallelizability
and inference efficiency. The proposed method can be widely applicable to
various SSM types like RetNet and Mamba. With similar model size, DenseSSM
achieves significant improvements, exemplified by DenseRetNet outperforming the
original RetNet with up to 5% accuracy improvement on public benchmarks.
65e782203b2aca8cd1e3c8ae
Wei He
65e782203b2aca8cd1e3c8af
Kai Han
claimed_verified
2024-03-07T09:15:25.683Z
65e52e7d27dc8aa470a640e3
/avatars/022a179d14de29b9ab9d96fcc85aa264.svg
hankai
user
hankaixyz
65e782203b2aca8cd1e3c8b0
Yehui Tang
admin_assigned
2024-03-05T21:00:10.881Z
64d5deb154bb9eb704f83122
/avatars/86ce09bcca903319051e2307581a43f4.svg
Yehui Tang
user
tangyehui
65e782203b2aca8cd1e3c8b1
Chengcheng Wang
65e782203b2aca8cd1e3c8b2
Yujie Yang
admin_assigned
2024-03-05T21:00:57.559Z
654d989f636d5f1b88a734f1
/avatars/f72e6c44b1c47b4a125370a7bf6e52c7.svg
Yujie Yang
user
EvanYYJ
65e782203b2aca8cd1e3c8b3
Tianyu Guo
admin_assigned
2024-03-05T21:01:04.778Z
62dd692c26b500df9124b2eb
/avatars/a490194a2839904e9497388bacf00c10.svg
Tianyu Guo
user
tiguo66
65e782203b2aca8cd1e3c8b4
Yunhe Wang
admin_assigned
2024-03-05T21:01:10.813Z
658bdf7b925aadd43304f05c
/avatars/64d9e9dea27c376c3bc7b2a54efc2a46.svg
Yunhe Wang
user
MightyCrane
DenseMamba: State Space Models with Dense Hidden Connection for
  Efficient Large Language Models
Large language models (LLMs) face a daunting challenge due to the excessive
computational and memory requirements of the commonly used Transformer
architecture. While state space model (SSM) is a new type of foundational
network architecture offering lower computational complexity, their performance
has yet to fully rival that of Transformers. This paper introduces DenseSSM, a
novel approach to enhance the flow of hidden information between layers in
SSMs. By selectively integrating shallowlayer hidden states into deeper layers,
DenseSSM retains fine-grained information crucial for the final output. Dense
connections enhanced DenseSSM still maintains the training parallelizability
and inference efficiency. The proposed method can be widely applicable to
various SSM types like RetNet and Mamba. With similar model size, DenseSSM
achieves significant improvements, exemplified by DenseRetNet outperforming the
original RetNet with up to 5% accuracy improvement on public benchmarks.
DenseSSM enhances state space model architectures by integrating shallow-layer hidden states into deeper layers, improving performance without increasing model size.
Transformers state space model (SSM) DenseSSM hidden information training parallelizability inference efficiency RetNet Mamba DenseRetNet