BlockFFN: Towards End-Side Acceleration-Friendly Mixture-of-Experts with
  Chunk-Level Activation Sparsity
To alleviate the computational burden of large language models (LLMs),
architectures with activation sparsity, represented by mixture-of-experts
(MoE), have attracted increasing attention. However, the non-differentiable and
inflexible routing of vanilla MoE hurts model performance. Moreover, while each
token activates only a few parameters, these sparsely-activated architectures
exhibit low chunk-level sparsity, indicating that the union of multiple
consecutive tokens activates a large ratio of parameters. Such a sparsity
pattern is unfriendly for acceleration under low-resource conditions (e.g.,
end-side devices) and incompatible with mainstream acceleration techniques
(e.g., speculative decoding). To address these challenges, we introduce a novel
MoE architecture, BlockFFN, as well as its efficient training and deployment
techniques. Specifically, we use a router integrating ReLU activation and
RMSNorm for differentiable and flexible routing. Next, to promote both
token-level sparsity (TLS) and chunk-level sparsity (CLS), CLS-aware training
objectives are designed, making BlockFFN more acceleration-friendly. Finally,
we implement efficient acceleration kernels, combining activation sparsity and
speculative decoding for the first time. The experimental results demonstrate
the superior performance of BlockFFN over other MoE baselines, achieving over
80% TLS and 70% 8-token CLS. Our kernels achieve up to 3.67times speedup on
real end-side devices than dense models. All codes and checkpoints are
available publicly (https://github.com/thunlp/BlockFFN).
68747a17257d4f04353702ef
Chenyang Song
claimed_verified
2025-07-15T19:10:37.056Z
64c09684e56520a63d35ec87
/avatars/86a338e8d122a66a94143bbb9bf3ebf8.svg
Chenyang Song
user
Raincleared
68747a17257d4f04353702f0
Weilin Zhao
68747a17257d4f04353702f1
Xu Han
68747a17257d4f04353702f2
Chaojun Xiao
68747a17257d4f04353702f3
Yingfa Chen
claimed_verified
2025-07-21T08:30:32.606Z
6144e4667f2544bb450787b2
https://cdn-avatars.huggingface.co/v1/production/uploads/6144e4667f2544bb450787b2/5wSDDqJbI4TGtBLP9IvjY.png
Yingfa Chen
user
chen-yingfa
68747a17257d4f04353702f4
Yuxuan Li
68747a17257d4f04353702f5
Zhiyuan Liu
68747a17257d4f04353702f6
Maosong Sun
BlockFFN: Towards End-Side Acceleration-Friendly Mixture-of-Experts with
  Chunk-Level Activation Sparsity
To alleviate the computational burden of large language models (LLMs),
architectures with activation sparsity, represented by mixture-of-experts
(MoE), have attracted increasing attention. However, the non-differentiable and
inflexible routing of vanilla MoE hurts model performance. Moreover, while each
token activates only a few parameters, these sparsely-activated architectures
exhibit low chunk-level sparsity, indicating that the union of multiple
consecutive tokens activates a large ratio of parameters. Such a sparsity
pattern is unfriendly for acceleration under low-resource conditions (e.g.,
end-side devices) and incompatible with mainstream acceleration techniques
(e.g., speculative decoding). To address these challenges, we introduce a novel
MoE architecture, BlockFFN, as well as its efficient training and deployment
techniques. Specifically, we use a router integrating ReLU activation and
RMSNorm for differentiable and flexible routing. Next, to promote both
token-level sparsity (TLS) and chunk-level sparsity (CLS), CLS-aware training
objectives are designed, making BlockFFN more acceleration-friendly. Finally,
we implement efficient acceleration kernels, combining activation sparsity and
speculative decoding for the first time. The experimental results demonstrate
the superior performance of BlockFFN over other MoE baselines, achieving over
80% TLS and 70% 8-token CLS. Our kernels achieve up to 3.67times speedup on
real end-side devices than dense models. All codes and checkpoints are
available publicly (https://github.com/thunlp/BlockFFN).
A novel MoE architecture, BlockFFN, with differentiable routing and CLS-aware training objectives, improves sparsity and acceleration for large language models on end-side devices.
mixture-of-experts MoE activation sparsity ReLU activation RMSNorm token-level sparsity chunk-level sparsity CLS-aware training speculative decoding BlockFFN