BitNet a4.8: 4-bit Activations for 1-bit LLMs
Recent research on the 1-bit Large Language Models (LLMs), such as BitNet
b1.58, presents a promising direction for reducing the inference cost of LLMs
while maintaining their performance. In this work, we introduce BitNet a4.8,
enabling 4-bit activations for 1-bit LLMs. BitNet a4.8 employs a hybrid
quantization and sparsification strategy to mitigate the quantization errors
introduced by the outlier channels. Specifically, we utilize 4-bit activations
for inputs to the attention and feed-forward network layers, while sparsifying
intermediate states followed with 8-bit quantization. Extensive experiments
demonstrate that BitNet a4.8 achieves performance comparable to BitNet b1.58
with equivalent training costs, while being faster in inference with enabling
4-bit (INT4/FP4) kernels. Additionally, BitNet a4.8 activates only 55% of
parameters and supports 3-bit KV cache, further enhancing the efficiency of
large-scale LLM deployment and inference.
672daab4c4c5c4fce10738fc
Hongyu Wang
claimed_verified
2024-11-15T09:29:25.856Z
63f71771d36951307fcb4dcd
https://cdn-avatars.huggingface.co/v1/production/uploads/63f71771d36951307fcb4dcd/1c-GSQmSSuDXyVWjxZFy_.jpeg
Hongyu Wang
user
hongyuw
672daab4c4c5c4fce10738fd
Shuming Ma
admin_assigned
2024-12-04T12:27:15.986Z
613f07f40153aafa379775f2
/avatars/3965175b320d753d9a5ccb0c7d9298a4.svg
Shuming Ma
user
shumingma
672daab4c4c5c4fce10738fe
Furu Wei
admin_assigned
2024-12-04T12:27:09.585Z
6368c512fbfe97c16a40baba
/avatars/1c23bc7c0b6d9225699ce27647623d7a.svg
Furu Wei
user
thegenerality
BitNet a4.8: 4-bit Activations for 1-bit LLMs
Recent research on the 1-bit Large Language Models (LLMs), such as BitNet
b1.58, presents a promising direction for reducing the inference cost of LLMs
while maintaining their performance. In this work, we introduce BitNet a4.8,
enabling 4-bit activations for 1-bit LLMs. BitNet a4.8 employs a hybrid
quantization and sparsification strategy to mitigate the quantization errors
introduced by the outlier channels. Specifically, we utilize 4-bit activations
for inputs to the attention and feed-forward network layers, while sparsifying
intermediate states followed with 8-bit quantization. Extensive experiments
demonstrate that BitNet a4.8 achieves performance comparable to BitNet b1.58
with equivalent training costs, while being faster in inference with enabling
4-bit (INT4/FP4) kernels. Additionally, BitNet a4.8 activates only 55% of
parameters and supports 3-bit KV cache, further enhancing the efficiency of
large-scale LLM deployment and inference.
BitNet a4.8 enhances the efficiency of large language models through 4-bit quantization and sparsification, achieving equivalent performance to BitNet b1.58 with reduced inference costs.
1-bit Large Language Models (LLMs) BitNet b1.58 BitNet a4.8 hybrid quantization sparsification quantization errors 4-bit activations attention layers feed-forward network layers 8-bit quantization INT4/FP4 kernels 3-bit KV cache