Addition is All You Need for Energy-efficient Language Models
Large neural networks spend most computation on floating point tensor
multiplications. In this work, we find that a floating point multiplier can be
approximated by one integer adder with high precision. We propose the
linear-complexity multiplication L-Mul algorithm that approximates floating
point number multiplication with integer addition operations. The new algorithm
costs significantly less computation resource than 8-bit floating point
multiplication but achieves higher precision. Compared to 8-bit floating point
multiplications, the proposed method achieves higher precision but consumes
significantly less bit-level computation. Since multiplying floating point
numbers requires substantially higher energy compared to integer addition
operations, applying the L-Mul operation in tensor processing hardware can
potentially reduce 95% energy cost by element-wise floating point tensor
multiplications and 80% energy cost of dot products. We calculated the
theoretical error expectation of L-Mul, and evaluated the algorithm on a wide
range of textual, visual, and symbolic tasks, including natural language
understanding, structural reasoning, mathematics, and commonsense question
answering. Our numerical analysis experiments agree with the theoretical error
estimation, which indicates that L-Mul with 4-bit mantissa achieves comparable
precision as float8_e4m3 multiplications, and L-Mul with 3-bit mantissa
outperforms float8_e5m2. Evaluation results on popular benchmarks show that
directly applying L-Mul to the attention mechanism is almost lossless. We
further show that replacing all floating point multiplications with 3-bit
mantissa L-Mul in a transformer model achieves equivalent precision as using
float8_e4m3 as accumulation precision in both fine-tuning and inference.
66fc9efa6d3e4a039730c68c
Hongyin Luo
admin_assigned
2024-10-03T09:38:09.909Z
62bc495016897fa76cdcda18
https://cdn-avatars.huggingface.co/v1/production/uploads/1656506661693-noauth.jpeg
Hongyin Luo
user
luohy
66fc9efa6d3e4a039730c68d
Wei Sun
Addition is All You Need for Energy-efficient Language Models
Large neural networks spend most computation on floating point tensor
multiplications. In this work, we find that a floating point multiplier can be
approximated by one integer adder with high precision. We propose the
linear-complexity multiplication L-Mul algorithm that approximates floating
point number multiplication with integer addition operations. The new algorithm
costs significantly less computation resource than 8-bit floating point
multiplication but achieves higher precision. Compared to 8-bit floating point
multiplications, the proposed method achieves higher precision but consumes
significantly less bit-level computation. Since multiplying floating point
numbers requires substantially higher energy compared to integer addition
operations, applying the L-Mul operation in tensor processing hardware can
potentially reduce 95% energy cost by element-wise floating point tensor
multiplications and 80% energy cost of dot products. We calculated the
theoretical error expectation of L-Mul, and evaluated the algorithm on a wide
range of textual, visual, and symbolic tasks, including natural language
understanding, structural reasoning, mathematics, and commonsense question
answering. Our numerical analysis experiments agree with the theoretical error
estimation, which indicates that L-Mul with 4-bit mantissa achieves comparable
precision as float8_e4m3 multiplications, and L-Mul with 3-bit mantissa
outperforms float8_e5m2. Evaluation results on popular benchmarks show that
directly applying L-Mul to the attention mechanism is almost lossless. We
further show that replacing all floating point multiplications with 3-bit
mantissa L-Mul in a transformer model achieves equivalent precision as using
float8_e4m3 as accumulation precision in both fine-tuning and inference.
A new integer addition-based algorithm reduces energy consumption in floating point tensor multiplications with comparable precision across various tasks.
floating point multiplier integer adder linear-complexity multiplication L-Mul algorithm tensor processing hardware error expectation natural language understanding structural reasoning mathematics commonsense question answering attention mechanism transformer model fine-tuning inference