Turbo Sparse: Achieving LLM SOTA Performance with Minimal Activated
  Parameters
Exploiting activation sparsity is a promising approach to significantly
accelerating the inference process of large language models (LLMs) without
compromising performance. However, activation sparsity is determined by
activation functions, and commonly used ones like SwiGLU and GeGLU exhibit
limited sparsity. Simply replacing these functions with ReLU fails to achieve
sufficient sparsity. Moreover, inadequate training data can further increase
the risk of performance degradation. To address these challenges, we propose a
novel dReLU function, which is designed to improve LLM activation sparsity,
along with a high-quality training data mixture ratio to facilitate effective
sparsification. Additionally, we leverage sparse activation patterns within the
Feed-Forward Network (FFN) experts of Mixture-of-Experts (MoE) models to
further boost efficiency. By applying our neuron sparsification method to the
Mistral and Mixtral models, only 2.5 billion and 4.3 billion parameters are
activated per inference iteration, respectively, while achieving even more
powerful model performance. Evaluation results demonstrate that this sparsity
achieves a 2-5x decoding speedup. Remarkably, on mobile phones, our
TurboSparse-Mixtral-47B achieves an inference speed of 11 tokens per second.
Our models are available at https://huggingface.co/PowerInfer
666a75d93536e9a956c5960c
Yixin Song
claimed_verified
2024-06-13T13:46:44.174Z
642924f916d4d8293c93af08
https://cdn-avatars.huggingface.co/v1/production/uploads/642924f916d4d8293c93af08/Olp4zFPgGu2E6mFFPXfQO.jpeg
Yixin Song
user
yixinsong
666a75d93536e9a956c5960d
Haotong Xie
666a75d93536e9a956c5960e
Zhengyan Zhang
admin_assigned
2024-06-13T20:03:31.823Z
65654ed2219af7f841640f27
/avatars/e6904b3479fc5e65ea1f752919ca8290.svg
Zhengyan Zhang
user
ZhengyanZhang
666a75d93536e9a956c5960f
Bo Wen
666a75d93536e9a956c59610
Li Ma
666a75d93536e9a956c59611
Zeyu Mi
claimed_verified
2024-06-13T09:30:32.510Z
6565a74575968d50cc1e51d3
/avatars/cc50e33b4e275aff2c017b7a2d7830e4.svg
Zeyu Mi
user
yzmizeyu
666a75d93536e9a956c59612
Haibo Chen
Turbo Sparse: Achieving LLM SOTA Performance with Minimal Activated
  Parameters
Exploiting activation sparsity is a promising approach to significantly
accelerating the inference process of large language models (LLMs) without
compromising performance. However, activation sparsity is determined by
activation functions, and commonly used ones like SwiGLU and GeGLU exhibit
limited sparsity. Simply replacing these functions with ReLU fails to achieve
sufficient sparsity. Moreover, inadequate training data can further increase
the risk of performance degradation. To address these challenges, we propose a
novel dReLU function, which is designed to improve LLM activation sparsity,
along with a high-quality training data mixture ratio to facilitate effective
sparsification. Additionally, we leverage sparse activation patterns within the
Feed-Forward Network (FFN) experts of Mixture-of-Experts (MoE) models to
further boost efficiency. By applying our neuron sparsification method to the
Mistral and Mixtral models, only 2.5 billion and 4.3 billion parameters are
activated per inference iteration, respectively, while achieving even more
powerful model performance. Evaluation results demonstrate that this sparsity
achieves a 2-5x decoding speedup. Remarkably, on mobile phones, our
TurboSparse-Mixtral-47B achieves an inference speed of 11 tokens per second.
Our models are available at https://huggingface.co/PowerInfer
A novel activation function (dReLU) and sparsification method improve the efficiency and speed of large language models without sacrificing performance.
activation sparsity SwiGLU GeGLU ReLU dReLU Feed-Forward Network (FFN) Mixture-of-Experts (MoE) neuron sparsification Mistral Mixtral