The Era of 1-bit LLMs: All Large Language Models are in 1.58 Bits
Recent research, such as BitNet, is paving the way for a new era of 1-bit
Large Language Models (LLMs). In this work, we introduce a 1-bit LLM variant,
namely BitNet b1.58, in which every single parameter (or weight) of the LLM is
ternary {-1, 0, 1}. It matches the full-precision (i.e., FP16 or BF16)
Transformer LLM with the same model size and training tokens in terms of both
perplexity and end-task performance, while being significantly more
cost-effective in terms of latency, memory, throughput, and energy consumption.
More profoundly, the 1.58-bit LLM defines a new scaling law and recipe for
training new generations of LLMs that are both high-performance and
cost-effective. Furthermore, it enables a new computation paradigm and opens
the door for designing specific hardware optimized for 1-bit LLMs.
65ded4ff33599459308af370
Shuming Ma
admin_assigned
2024-02-28T11:01:44.539Z
613f07f40153aafa379775f2
/avatars/3965175b320d753d9a5ccb0c7d9298a4.svg
Shuming Ma
user
shumingma
65ded4ff33599459308af371
Hongyu Wang
claimed_verified
2024-02-28T11:41:14.755Z
63f71771d36951307fcb4dcd
https://cdn-avatars.huggingface.co/v1/production/uploads/63f71771d36951307fcb4dcd/1c-GSQmSSuDXyVWjxZFy_.jpeg
Hongyu Wang
user
hongyuw
65ded4ff33599459308af372
Lingxiao Ma
admin_assigned
2024-09-18T13:16:16.361Z
63d9074f89a9683b46d1c73d
/avatars/6dc070c7618f24b76d9df9882fcd5aed.svg
Lingxiao Ma
user
xysmlx
65ded4ff33599459308af373
Lei Wang
claimed_verified
2024-07-03T07:42:25.603Z
646c7b3849a37754bdeefe8e
https://cdn-avatars.huggingface.co/v1/production/uploads/646c7b3849a37754bdeefe8e/tejirZfCHm6EqhXOdnMUq.jpeg
Lei
user
LeiWang1999
65ded4ff33599459308af374
Wenhui Wang
65ded4ff33599459308af375
Shaohan Huang
admin_assigned
2024-02-28T11:06:04.566Z
632bd2f72d6a805eeb4bc601
/avatars/6e1533e8a599f3068290aa69ac82cab7.svg
HUANG SHAOHAN
user
buaahsh
65ded4ff33599459308af376
Li Dong
claimed_verified
2024-03-01T15:53:48.354Z
5df85abada6d0311fd3d5408
/avatars/2331cf703c1b5d3a62e2050b1a6eb108.svg
Li Dong
user
unilm
65ded4ff33599459308af377
Ruiping Wang
admin_assigned
2024-02-28T11:05:07.170Z
64fa7e742d20ced4e9ecd835
/avatars/52db257f8600562905be61d16b78a4f4.svg
ruipingwang
user
RuipingWang1986
65ded4ff33599459308af378
Jilong Xue
admin_assigned
2024-02-28T11:04:47.786Z
632bfc0e4a4991e7114f2a83
/avatars/19f8154a30d4787c5ebdf9657e37ea12.svg
Jilong Xue
user
jlxue
65ded4ff33599459308af379
Furu Wei
admin_assigned
2024-02-28T11:03:55.843Z
6368c512fbfe97c16a40baba
/avatars/1c23bc7c0b6d9225699ce27647623d7a.svg
Furu Wei
user
thegenerality
The Era of 1-bit LLMs: All Large Language Models are in 1.58 Bits
Recent research, such as BitNet, is paving the way for a new era of 1-bit
Large Language Models (LLMs). In this work, we introduce a 1-bit LLM variant,
namely BitNet b1.58, in which every single parameter (or weight) of the LLM is
ternary {-1, 0, 1}. It matches the full-precision (i.e., FP16 or BF16)
Transformer LLM with the same model size and training tokens in terms of both
perplexity and end-task performance, while being significantly more
cost-effective in terms of latency, memory, throughput, and energy consumption.
More profoundly, the 1.58-bit LLM defines a new scaling law and recipe for
training new generations of LLMs that are both high-performance and
cost-effective. Furthermore, it enables a new computation paradigm and opens
the door for designing specific hardware optimized for 1-bit LLMs.
A 1-bit LLM variant, BitNet b1.58, achieves comparable performance to full-precision models with reduced computational costs and introduces new scaling laws and hardware design opportunities.
BitNet 1-bit LLM ternary Transformer LLM perplexity end-task performance latency memory throughput energy consumption scaling law hardware optimization