Binary and Ternary Natural Language Generation
Ternary and binary neural networks enable multiplication-free computation and
promise multiple orders of magnitude efficiency gains over full-precision
networks if implemented on specialized hardware. However, since both the
parameter and the output space are highly discretized, such networks have
proven very difficult to optimize. The difficulties are compounded for the
class of transformer text generation models due to the sensitivity of the
attention operation to quantization and the noise-compounding effects of
autoregressive decoding in the high-cardinality output space. We approach the
problem with a mix of statistics-based quantization for the weights and elastic
quantization of the activations and demonstrate the first ternary and binary
transformer models on the downstream tasks of summarization and machine
translation. Our ternary BART base achieves an R1 score of 41 on the
CNN/DailyMail benchmark, which is merely 3.9 points behind the full model while
being 16x more efficient. Our binary model, while less accurate, achieves a
highly non-trivial score of 35.6. For machine translation, we achieved BLEU
scores of 21.7 and 17.6 on the WMT16 En-Ro benchmark, compared with a full
precision mBART model score of 26.8. We also compare our approach in the 8-bit
activation setting, where our ternary and even binary weight models can match
or outperform the best existing 8-bit weight models in the literature. Our code
and models are available at:
https://github.com/facebookresearch/Ternary_Binary_Transformer
647ebce9becb41a27297d7f6
Zechun Liu
claimed_verified
2023-06-06T17:23:48.383Z
620ae55fe190de4b8f27ba9d
/avatars/aaa2d0eadbcf96c2eb9059e3d73c2760.svg
Liu
user
Zechun
647ebce9becb41a27297d7f7
Barlas Oguz
extracted_pending
2025-02-25T00:32:56.632Z
6798100acbb6655a3cbca13b
/avatars/c2a53c37d39753dc53e12a3349b026a0.svg
Barlas Oguz
user
borguz
647ebce9becb41a27297d7f8
Aasish Pappu
647ebce9becb41a27297d7f9
Yangyang Shi
extracted_pending
2024-11-01T19:54:26.400Z
638fca982ddd69e70b8bdb8d
/avatars/58ee470ca8cd3f0ae01bcaad33911755.svg
Yangyang Shi
user
yyshi
647ebce9becb41a27297d7fa
Raghuraman Krishnamoorthi
Binary and Ternary Natural Language Generation
Ternary and binary neural networks enable multiplication-free computation and
promise multiple orders of magnitude efficiency gains over full-precision
networks if implemented on specialized hardware. However, since both the
parameter and the output space are highly discretized, such networks have
proven very difficult to optimize. The difficulties are compounded for the
class of transformer text generation models due to the sensitivity of the
attention operation to quantization and the noise-compounding effects of
autoregressive decoding in the high-cardinality output space. We approach the
problem with a mix of statistics-based quantization for the weights and elastic
quantization of the activations and demonstrate the first ternary and binary
transformer models on the downstream tasks of summarization and machine
translation. Our ternary BART base achieves an R1 score of 41 on the
CNN/DailyMail benchmark, which is merely 3.9 points behind the full model while
being 16x more efficient. Our binary model, while less accurate, achieves a
highly non-trivial score of 35.6. For machine translation, we achieved BLEU
scores of 21.7 and 17.6 on the WMT16 En-Ro benchmark, compared with a full
precision mBART model score of 26.8. We also compare our approach in the 8-bit
activation setting, where our ternary and even binary weight models can match
or outperform the best existing 8-bit weight models in the literature. Our code
and models are available at:
https://github.com/facebookresearch/Ternary_Binary_Transformer
Ternary and binary transformer models achieve significant efficiency gains with minor accuracy drops compared to full-precision counterparts, particularly in summarization and machine translation tasks.
ternary neural networks binary neural networks multiplication-free computation parameter quantization elastic quantization activations quantization transformer text generation attention operation autoregressive decoding high-cardinality output space R1 score BLEU scores CNN/DailyMail benchmark WMT16 En-Ro benchmark mBART model