OneBit: Towards Extremely Low-bit Large Language Models
Model quantification uses low bit-width values to represent the weight
matrices of models, which is a promising approach to reduce both storage and
computational overheads of deploying highly anticipated LLMs. However, existing
quantization methods suffer severe performance degradation when the bit-width
is extremely reduced, and thus focus on utilizing 4-bit or 8-bit values to
quantize models. This paper boldly quantizes the weight matrices of LLMs to
1-bit, paving the way for the extremely low bit-width deployment of LLMs. For
this target, we introduce a 1-bit quantization-aware training (QAT) framework
named OneBit, including a novel 1-bit parameter representation method to better
quantize LLMs as well as an effective parameter initialization method based on
matrix decomposition to improve the convergence speed of the QAT framework.
Sufficient experimental results indicate that OneBit achieves good performance
(at least 83% of the non-quantized performance) with robust training processes
when only using 1-bit weight matrices.
65d43c4d9f087171b8fa0474
Yuzhuang Xu
admin_assigned
2024-02-20T10:01:49.435Z
64b89a5403124195cd840b91
/avatars/4141144d4df2ca652931c8c15bd24b7c.svg
Yuzhuang Xu
user
Yuzhuang
65d43c4d9f087171b8fa0475
Xu Han
65d43c4d9f087171b8fa0476
Zonghan Yang
claimed_verified
2024-02-21T08:30:32.346Z
635795f24fba057662cdf2e7
/avatars/07642c7cfd9b57a0f8382268b009cf6f.svg
Zonghan Yang
user
minicheshire
65d43c4d9f087171b8fa0477
Shuo Wang
65d43c4d9f087171b8fa0478
Qingfu Zhu
admin_assigned
2024-02-20T10:03:26.373Z
649ce641dd87dd9ef76e3628
/avatars/d5fdeb9e2131ada258b776fec95eb00b.svg
zhu
user
qingfuhf
65d43c4d9f087171b8fa0479
Zhiyuan Liu
claimed_verified
2025-06-02T08:00:50.567Z
6472f1f7485a7c8e1bcf507b
/avatars/79dca137583875191ea883b193b630b7.svg
Zhiyuan Liu
user
zibuyu9
65d43c4d9f087171b8fa047a
Weidong Liu
65d43c4d9f087171b8fa047b
Wanxiang Che
admin_assigned
2024-02-20T10:05:05.609Z
6325418ae5d49a0d40f66176
/avatars/59a0e29ce09e9113edc53cebb728b099.svg
Wanxiang Che
user
wanxiangche
OneBit: Towards Extremely Low-bit Large Language Models
Model quantification uses low bit-width values to represent the weight
matrices of models, which is a promising approach to reduce both storage and
computational overheads of deploying highly anticipated LLMs. However, existing
quantization methods suffer severe performance degradation when the bit-width
is extremely reduced, and thus focus on utilizing 4-bit or 8-bit values to
quantize models. This paper boldly quantizes the weight matrices of LLMs to
1-bit, paving the way for the extremely low bit-width deployment of LLMs. For
this target, we introduce a 1-bit quantization-aware training (QAT) framework
named OneBit, including a novel 1-bit parameter representation method to better
quantize LLMs as well as an effective parameter initialization method based on
matrix decomposition to improve the convergence speed of the QAT framework.
Sufficient experimental results indicate that OneBit achieves good performance
(at least 83% of the non-quantized performance) with robust training processes
when only using 1-bit weight matrices.
OneBit, a 1-bit quantization-aware training framework, allows LLMs to achieve robust performance with 1-bit weight matrices, reducing storage and computational overhead.
model quantification low bit-width weight matrices LLMs quantization methods 1-bit quantization-aware training (QAT) parameter representation matrix decomposition performance convergence speed