An Efficient Sparse Inference Software Accelerator for Transformer-based
  Language Models on CPUs
In recent years, Transformer-based language models have become the standard
approach for natural language processing tasks. However, stringent throughput
and latency requirements in industrial applications are limiting their
adoption. To mitigate the gap, model compression techniques such as structured
pruning are being used to improve inference efficiency. However, most existing
neural network inference runtimes lack adequate support for structured
sparsity. In this paper, we propose an efficient sparse deep learning inference
software stack for Transformer-based language models where the weights are
pruned with constant block size. Our sparse software accelerator leverages
Intel Deep Learning Boost to maximize the performance of sparse matrix - dense
matrix multiplication (commonly abbreviated as SpMM) on CPUs. Our SpMM kernel
outperforms the existing sparse libraries (oneMKL, TVM, and LIBXSMM) by an
order of magnitude on a wide range of GEMM shapes under 5 representative
sparsity ratios (70%, 75%, 80%, 85%, 90%). Moreover, our SpMM kernel shows up
to 5x speedup over dense GEMM kernel of oneDNN, a well-optimized dense library
widely used in industry. We apply our sparse accelerator on widely-used
Transformer-based language models including Bert-Mini, DistilBERT, Bert-Base,
and BERT-Large. Our sparse inference software shows up to 1.5x speedup over
Neural Magic's Deepsparse under same configurations on Xeon on Amazon Web
Services under proxy production latency constraints. We also compare our
solution with two framework-based inference solutions, ONNX Runtime and
PyTorch, and demonstrate up to 37x speedup over ONNX Runtime and 345x over
PyTorch on Xeon under the latency constraints. All the source code is publicly
available on Github: https://github.com/intel/intel-extension-for-transformers.
649e3821ed55df9fd24331b2
Haihao Shen
admin_assigned
2023-06-30T20:17:58.616Z
60ac3318e3de7c7440abb850
https://cdn-avatars.huggingface.co/v1/production/uploads/60ac3318e3de7c7440abb850/DkMrPBr6Ew_dS_c9kog4e.jpeg
Haihao Shen
user
Haihao
649e3821ed55df9fd24331b3
Hengyu Meng
admin_assigned
2023-06-30T20:18:13.773Z
63edfe1c5f1300034dd90a37
/avatars/fa742c6da029ce73b395d1a12a97753f.svg
Hyuno
user
HengyuMeng
649e3821ed55df9fd24331b4
Bo Dong
admin_assigned
2023-06-30T20:18:30.023Z
616e6906a6e9a95e08a09c9d
/avatars/2a8dece24cbc1779eb906d5582573c49.svg
Dong, Bo
user
BoDong
649e3821ed55df9fd24331b5
Zhe Wang
649e3821ed55df9fd24331b6
Ofir Zafrir
admin_assigned
2023-06-30T20:40:15.668Z
5f8907c65d083370c711f284
https://cdn-avatars.huggingface.co/v1/production/uploads/1616423186722-5f8907c65d083370c711f284.jpeg
Ofir Zafrir
user
ofirzaf
649e3821ed55df9fd24331b7
Yi Ding
649e3821ed55df9fd24331b8
Yu Luo
649e3821ed55df9fd24331b9
Hanwen Chang
649e3821ed55df9fd24331ba
Qun Gao
admin_assigned
2023-06-30T20:42:24.684Z
63079d748b054d85f299e1e4
/avatars/d4947439475dc81f2c9e9304382b6257.svg
Qun Gao
user
qgao007
649e3821ed55df9fd24331bb
Ziheng Wang
649e3821ed55df9fd24331bc
Guy Boudoukh
admin_assigned
2023-06-30T20:42:55.307Z
60abe77d08805574fcf076f2
/avatars/06df4ead5a2014480c128103b9862f98.svg
Guy Boudoukh
user
guybd
649e3821ed55df9fd24331bd
Moshe Wasserblat
admin_assigned
2023-06-30T20:43:12.249Z
6013c64031d84a3bb7650038
/avatars/e05ca715004b39e79472399f75010bda.svg
Moshe Wasserblat
user
moshew
An Efficient Sparse Inference Software Accelerator for Transformer-based
  Language Models on CPUs
In recent years, Transformer-based language models have become the standard
approach for natural language processing tasks. However, stringent throughput
and latency requirements in industrial applications are limiting their
adoption. To mitigate the gap, model compression techniques such as structured
pruning are being used to improve inference efficiency. However, most existing
neural network inference runtimes lack adequate support for structured
sparsity. In this paper, we propose an efficient sparse deep learning inference
software stack for Transformer-based language models where the weights are
pruned with constant block size. Our sparse software accelerator leverages
Intel Deep Learning Boost to maximize the performance of sparse matrix - dense
matrix multiplication (commonly abbreviated as SpMM) on CPUs. Our SpMM kernel
outperforms the existing sparse libraries (oneMKL, TVM, and LIBXSMM) by an
order of magnitude on a wide range of GEMM shapes under 5 representative
sparsity ratios (70%, 75%, 80%, 85%, 90%). Moreover, our SpMM kernel shows up
to 5x speedup over dense GEMM kernel of oneDNN, a well-optimized dense library
widely used in industry. We apply our sparse accelerator on widely-used
Transformer-based language models including Bert-Mini, DistilBERT, Bert-Base,
and BERT-Large. Our sparse inference software shows up to 1.5x speedup over
Neural Magic's Deepsparse under same configurations on Xeon on Amazon Web
Services under proxy production latency constraints. We also compare our
solution with two framework-based inference solutions, ONNX Runtime and
PyTorch, and demonstrate up to 37x speedup over ONNX Runtime and 345x over
PyTorch on Xeon under the latency constraints. All the source code is publicly
available on Github: https://github.com/intel/intel-extension-for-transformers.
A sparse inference software stack for Transformer-based language models significantly improves performance through efficient SpMM operations on CPUs, offering substantial speedups over existing frameworks and solutions.
Transformer-based language models structured pruning sparse deep learning inference SpMM Intel Deep Learning Boost oneMKL TVM LIBXSMM SpMM kernel Bert-Mini DistilBERT Bert-Base BERT-Large Neural Magic's Deepsparse ONNX Runtime PyTorch sparse matrix - dense matrix multiplication