SkipDecode: Autoregressive Skip Decoding with Batching and Caching for
  Efficient LLM Inference
Autoregressive large language models (LLMs) have made remarkable progress in
various natural language generation tasks. However, they incur high computation
cost and latency resulting from the autoregressive token-by-token generation.
To address this issue, several approaches have been proposed to reduce
computational cost using early-exit strategies. These strategies enable faster
text generation using reduced computation without applying the full computation
graph to each token. While existing token-level early exit methods show
promising results for online inference, they cannot be readily applied for
batch inferencing and Key-Value caching. This is because they have to wait
until the last token in a batch exits before they can stop computing. This
severely limits the practical application of such techniques. In this paper, we
propose a simple and effective token-level early exit method, SkipDecode,
designed to work seamlessly with batch inferencing and KV caching. It overcomes
prior constraints by setting up a singular exit point for every token in a
batch at each sequence position. It also guarantees a monotonic decrease in
exit points, thereby eliminating the need to recompute KV Caches for preceding
tokens. Rather than terminating computation prematurely as in prior works, our
approach bypasses lower to middle layers, devoting most of the computational
resources to upper layers, allowing later tokens to benefit from the compute
expenditure by earlier tokens. Our experimental results show that SkipDecode
can obtain 2x to 5x inference speedups with negligible regression across a
variety of tasks. This is achieved using OPT models of 1.3 billion and 6.7
billion parameters, all the while being directly compatible with batching and
KV caching optimization techniques.
64a76ab00c36529d75fef32c
Luciano Del Corro
admin_assigned
2023-07-11T08:03:57.586Z
63b81aace60862785afd8ca2
/avatars/f96ce78af6ad42514235bab811544789.svg
Luciano Del Corro
user
lucianodelcorro
64a76ab00c36529d75fef32d
Allie Del Giorno
admin_assigned
2023-07-11T08:03:36.527Z
63f5562471a5d395c721cd8e
/avatars/ac296b6017e86ea04c73803fe2c44433.svg
Allie Del Giorno
user
microallie
64a76ab00c36529d75fef32e
Sahaj Agarwal
64a76ab00c36529d75fef32f
Bin Yu
admin_assigned
2023-07-11T08:07:32.442Z
64679498374fe5728d4044d5
/avatars/78516e1c9532900cce19f81483855db8.svg
Bin Yu
user
binyu000
64a76ab00c36529d75fef330
Ahmed Awadallah
64a76ab00c36529d75fef331
Subhabrata Mukherjee
admin_assigned
2023-07-11T08:55:28.501Z
5fd2dbbd9e5a0a03458e1fc8
/avatars/d0868aef6c180db8610184396443731a.svg
Subhabrata Mukherjee
user
subho
SkipDecode: Autoregressive Skip Decoding with Batching and Caching for
  Efficient LLM Inference
Autoregressive large language models (LLMs) have made remarkable progress in
various natural language generation tasks. However, they incur high computation
cost and latency resulting from the autoregressive token-by-token generation.
To address this issue, several approaches have been proposed to reduce
computational cost using early-exit strategies. These strategies enable faster
text generation using reduced computation without applying the full computation
graph to each token. While existing token-level early exit methods show
promising results for online inference, they cannot be readily applied for
batch inferencing and Key-Value caching. This is because they have to wait
until the last token in a batch exits before they can stop computing. This
severely limits the practical application of such techniques. In this paper, we
propose a simple and effective token-level early exit method, SkipDecode,
designed to work seamlessly with batch inferencing and KV caching. It overcomes
prior constraints by setting up a singular exit point for every token in a
batch at each sequence position. It also guarantees a monotonic decrease in
exit points, thereby eliminating the need to recompute KV Caches for preceding
tokens. Rather than terminating computation prematurely as in prior works, our
approach bypasses lower to middle layers, devoting most of the computational
resources to upper layers, allowing later tokens to benefit from the compute
expenditure by earlier tokens. Our experimental results show that SkipDecode
can obtain 2x to 5x inference speedups with negligible regression across a
variety of tasks. This is achieved using OPT models of 1.3 billion and 6.7
billion parameters, all the while being directly compatible with batching and
KV caching optimization techniques.
SkipDecode, a token-level early exit method, enhances batch inferencing and KV caching in autoregressive LLMs, achieving significant speedups while maintaining performance.
autoregressive large language models early-exit strategies token-level early exit batch inferencing Key-Value caching SkipDecode sequence position computational resources upper layers inference speedups OPT models