BitNet Distillation
In this paper, we present BitNet Distillation (BitDistill), a lightweight
pipeline that fine-tunes off-the-shelf full-precision LLMs (e.g., Qwen) into
1.58-bit precision (i.e., ternary weights {-1, 0, 1}) for specific downstream
tasks, achieving strong task-specific performance with minimal computational
cost. Specifically, BitDistill incorporates three key techniques: the SubLN
module, as introduced in BitNet; multi-head attention distillation, based on
MiniLM; and continual pre-training, which serves as a crucial warm-up step to
mitigate the scalability issue of the performance gap between finetuned
full-precision and 1.58-bit LLMs on specific tasks. Experimental results show
that BitDistill achieves performance comparable to the full-precision
counterpart models across model size, while enabling up to 10x memory savings
and 2.65x faster inference on CPUs. Code is available at
https://github.com/microsoft/BitNet.
68f1b1f06e0bef323a68fde5
Xun Wu
68f1b1f06e0bef323a68fde6
Shaohan Huang
68f1b1f06e0bef323a68fde7
Wenhui Wang
68f1b1f06e0bef323a68fde8
Ting Song
68f1b1f06e0bef323a68fde9
Li Dong
68f1b1f06e0bef323a68fdea
Yan Xia
68f1b1f06e0bef323a68fdeb
Furu Wei
BitNet Distillation
In this paper, we present BitNet Distillation (BitDistill), a lightweight
pipeline that fine-tunes off-the-shelf full-precision LLMs (e.g., Qwen) into
1.58-bit precision (i.e., ternary weights {-1, 0, 1}) for specific downstream
tasks, achieving strong task-specific performance with minimal computational
cost. Specifically, BitDistill incorporates three key techniques: the SubLN
module, as introduced in BitNet; multi-head attention distillation, based on
MiniLM; and continual pre-training, which serves as a crucial warm-up step to
mitigate the scalability issue of the performance gap between finetuned
full-precision and 1.58-bit LLMs on specific tasks. Experimental results show
that BitDistill achieves performance comparable to the full-precision
counterpart models across model size, while enabling up to 10x memory savings
and 2.65x faster inference on CPUs. Code is available at
https://github.com/microsoft/BitNet.
BitNet Distillation fine-tunes large language models to 1.58-bit precision using SubLN, multi-head attention distillation, and continual pre-training, achieving comparable performance with significant memory and inference speed improvements.
BitNet Distillation BitDistill SubLN multi-head attention distillation continual pre-training LLMs Qwen ternary weights memory savings inference speed