Scaling TransNormer to 175 Billion Parameters
We present TransNormerLLM, the first linear attention-based Large Language
Model (LLM) that outperforms conventional softmax attention-based models in
terms of both accuracy and efficiency. TransNormerLLM evolves from the previous
linear attention architecture TransNormer by making advanced modifications that
include positional embedding, linear attention acceleration, gating mechanism,
tensor normalization, inference acceleration and stabilization. Specifically,
we use LRPE together with an exponential decay to avoid attention dilution
issues while allowing the model to retain global interactions between tokens.
Additionally, we propose Lightning Attention, a cutting-edge technique that
accelerates linear attention by more than twice in runtime and reduces memory
usage by a remarkable four times. To further enhance the performance of
TransNormer, we leverage a gating mechanism to smooth training and a new tensor
normalization scheme to accelerate the model, resulting in an impressive
acceleration of over 20%. Furthermore, we have developed a robust inference
algorithm that ensures numerical stability and consistent inference speed,
regardless of the sequence length, showcasing superior efficiency during both
training and inference stages. Scalability is at the heart of our model's
design, enabling seamless deployment on large-scale clusters and facilitating
expansion to even more extensive models, all while maintaining outstanding
performance metrics. Rigorous validation of our model design is achieved
through a series of comprehensive experiments on our self-collected corpus,
boasting a size exceeding 6TB and containing over 2 trillion tokens. To ensure
data quality and relevance, we implement a new self-cleaning strategy to filter
our collected data. Our pre-trained models will be released to foster community
advancements in efficient LLMs.
64c317511920bd982085c652
Zhen Qin
claimed_verified
2024-12-04T09:48:04.814Z
642e63a53c2cf43f6d6dc5ce
/avatars/dfd78c8d55485c22be6e616670a633e5.svg
zhenqin
user
Doreamonzzz
64c317511920bd982085c653
Dong Li
claimed_verified
2023-11-13T08:32:00.398Z
61cd713f78c1c6b489a231fc
/avatars/af261b85bbd10a1182372ffc459640b8.svg
Dong Li
user
liddalidd
64c317511920bd982085c654
Weigao Sun
admin_assigned
2023-07-28T08:18:48.288Z
6246bb33da617c00b48e4d92
/avatars/0304a9f6eb7f5dee4d933d03222f94e9.svg
Weigao Sun
user
weigao266
64c317511920bd982085c655
Weixuan Sun
admin_assigned
2023-07-28T08:19:08.964Z
640c4d2d536d9fe0f000b209
/avatars/419e59b78f7fc7f6b3ac343398d11ec1.svg
Weixuan Sun
user
weixuansun
64c317511920bd982085c656
Xuyang Shen
claimed_verified
2024-06-25T07:35:15.641Z
642e4d4d6748dd4f8eeb7732
/avatars/fd911e9143d1a7aedd21a7d611543fcc.svg
Xuyang Shen
user
Ryan1122
64c317511920bd982085c657
Xiaodong Han
admin_assigned
2023-07-28T08:20:11.394Z
64638bddc615cbc12447f9f1
/avatars/52910d1717451ff983f745322e5850dd.svg
Xiaodong Han
user
Hannnnnxd
64c317511920bd982085c658
Yunshen Wei
64c317511920bd982085c659
Baohong Lv
64c317511920bd982085c65a
Fei Yuan
claimed_verified
2024-03-25T09:45:13.208Z
65fed45b08d35929362dd651
https://cdn-avatars.huggingface.co/v1/production/uploads/65fed45b08d35929362dd651/KLMxsyRN6_HhCZP1iDw6K.png
FeiYuan
user
FeYuan
64c317511920bd982085c65b
Xiao Luo
claimed_verified
2025-06-17T13:07:46.115Z
612741a391de43c1101df014
/avatars/1461b1c7d3cedd91cea6cf3b0ecb14ae.svg
Rock Luo
user
windlx
64c317511920bd982085c65c
Yu Qiao
64c317511920bd982085c65d
Yiran Zhong
claimed_verified
2025-01-20T09:32:38.615Z
64c525e4d68946edad6c7067
/avatars/1b108661634af602717a4ab4b66a151f.svg
Yiran Zhong
user
IanZhong
Scaling TransNormer to 175 Billion Parameters
We present TransNormerLLM, the first linear attention-based Large Language
Model (LLM) that outperforms conventional softmax attention-based models in
terms of both accuracy and efficiency. TransNormerLLM evolves from the previous
linear attention architecture TransNormer by making advanced modifications that
include positional embedding, linear attention acceleration, gating mechanism,
tensor normalization, inference acceleration and stabilization. Specifically,
we use LRPE together with an exponential decay to avoid attention dilution
issues while allowing the model to retain global interactions between tokens.
Additionally, we propose Lightning Attention, a cutting-edge technique that
accelerates linear attention by more than twice in runtime and reduces memory
usage by a remarkable four times. To further enhance the performance of
TransNormer, we leverage a gating mechanism to smooth training and a new tensor
normalization scheme to accelerate the model, resulting in an impressive
acceleration of over 20%. Furthermore, we have developed a robust inference
algorithm that ensures numerical stability and consistent inference speed,
regardless of the sequence length, showcasing superior efficiency during both
training and inference stages. Scalability is at the heart of our model's
design, enabling seamless deployment on large-scale clusters and facilitating
expansion to even more extensive models, all while maintaining outstanding
performance metrics. Rigorous validation of our model design is achieved
through a series of comprehensive experiments on our self-collected corpus,
boasting a size exceeding 6TB and containing over 2 trillion tokens. To ensure
data quality and relevance, we implement a new self-cleaning strategy to filter
our collected data. Our pre-trained models will be released to foster community
advancements in efficient LLMs.
TransNormerLLM, a linear attention-based LLM, outperforms softmax attention models through advanced modifications like positional embedding, lightning attention, gating mechanisms, and tensor normalization, achieving better accuracy and efficiency.
linear attention softmax attention TransNormer positional embedding LRPE exponential decay attention dilution Lightning Attention gating mechanism tensor normalization inference algorithm numerical stability sequence length scalability pre-trained models