Mixture of Scales: Memory-Efficient Token-Adaptive Binarization for
  Large Language Models
Binarization, which converts weight parameters to binary values, has emerged
as an effective strategy to reduce the size of large language models (LLMs).
However, typical binarization techniques significantly diminish linguistic
effectiveness of LLMs. To address this issue, we introduce a novel binarization
technique called Mixture of Scales (BinaryMoS). Unlike conventional methods,
BinaryMoS employs multiple scaling experts for binary weights, dynamically
merging these experts for each token to adaptively generate scaling factors.
This token-adaptive approach boosts the representational power of binarized
LLMs by enabling contextual adjustments to the values of binary weights.
Moreover, because this adaptive process only involves the scaling factors
rather than the entire weight matrix, BinaryMoS maintains compression
efficiency similar to traditional static binarization methods. Our experimental
results reveal that BinaryMoS surpasses conventional binarization techniques in
various natural language processing tasks and even outperforms 2-bit
quantization methods, all while maintaining similar model size to static
binarization techniques.
6672682a6d27e245f657825a
Dongwon Jo
claimed_verified
2024-06-19T07:37:26.386Z
639ffbc6beb95d698de9640d
/avatars/7ef1aaadd5b378d00e17dc548e42cb7e.svg
Dongwon Jo
user
dongwonjo
6672682a6d27e245f657825b
Taesu Kim
admin_assigned
2024-06-20T10:00:19.449Z
64dc388069d21c567b0794b0
/avatars/795ab1b7f970df3b589f6c1ca75df9ad.svg
Taesu Kim
user
tae-su-kim
6672682a6d27e245f657825c
Yulhwa Kim
admin_assigned
2024-06-20T10:00:40.017Z
6566ddb96af53c602f80b1e2
/avatars/403c8e486115920e50867b6462ddfd99.svg
Yulhwa Kim
user
YulhwaKim
6672682a6d27e245f657825d
Jae-Joon Kim
Mixture of Scales: Memory-Efficient Token-Adaptive Binarization for
  Large Language Models
Binarization, which converts weight parameters to binary values, has emerged
as an effective strategy to reduce the size of large language models (LLMs).
However, typical binarization techniques significantly diminish linguistic
effectiveness of LLMs. To address this issue, we introduce a novel binarization
technique called Mixture of Scales (BinaryMoS). Unlike conventional methods,
BinaryMoS employs multiple scaling experts for binary weights, dynamically
merging these experts for each token to adaptively generate scaling factors.
This token-adaptive approach boosts the representational power of binarized
LLMs by enabling contextual adjustments to the values of binary weights.
Moreover, because this adaptive process only involves the scaling factors
rather than the entire weight matrix, BinaryMoS maintains compression
efficiency similar to traditional static binarization methods. Our experimental
results reveal that BinaryMoS surpasses conventional binarization techniques in
various natural language processing tasks and even outperforms 2-bit
quantization methods, all while maintaining similar model size to static
binarization techniques.
A novel binarization technique called Mixture of Scales (BinaryMoS) is introduced to enhance linguistic effectiveness while maintaining model size and compression efficiency in large language models.
binarization large language models Mixture of Scales BinaryMoS scaling experts token-adaptive representational power binary weights compression efficiency static binarization 2-bit quantization