LayerSkip: Enabling Early Exit Inference and Self-Speculative Decoding
We present LayerSkip, an end-to-end solution to speed-up inference of large
language models (LLMs). First, during training we apply layer dropout, with low
dropout rates for earlier layers and higher dropout rates for later layers, and
an early exit loss where all transformer layers share the same exit. Second,
during inference, we show that this training recipe increases the accuracy of
early exit at earlier layers, without adding any auxiliary layers or modules to
the model. Third, we present a novel self-speculative decoding solution where
we exit at early layers and verify and correct with remaining layers of the
model. Our proposed self-speculative decoding approach has less memory
footprint than other speculative decoding approaches and benefits from shared
compute and activations of the draft and verification stages. We run
experiments on different Llama model sizes on different types of training:
pretraining from scratch, continual pretraining, finetuning on specific data
domain, and finetuning on specific task. We implement our inference solution
and show speedups of up to 2.16x on summarization for CNN/DM documents, 1.82x
on coding, and 2.0x on TOPv2 semantic parsing task. We open source our code and
checkpoints at https://github.com/facebookresearch/LayerSkip.
662b1842ef7a4675bdfd9323
Mostafa Elhoushi
extracted_confirmed
2024-04-26T19:58:42.088Z
63c9725ebedad7e2bf160bdc
https://cdn-avatars.huggingface.co/v1/production/uploads/63c9725ebedad7e2bf160bdc/wzPuyhOXCYBNGwZDshbnL.jpeg
Mostafa Elhoushi
user
melhoushi
662b1842ef7a4675bdfd9324
Akshat Shrivastava
extracted_pending
2024-04-26T02:58:11.458Z
64b6d105f616b195a39f462a
/avatars/f33805c3e42e476a135c3cf869cc423a.svg
Akshat
user
akshats07
662b1842ef7a4675bdfd9325
Diana Liskovich
662b1842ef7a4675bdfd9326
Basil Hosmer
admin_assigned
2024-04-26T07:47:01.676Z
631183a88894270062a2ea36
/avatars/5da43a25768f5b677bbd02bd955210e7.svg
Basil Hosmer
user
bhosmer
662b1842ef7a4675bdfd9327
Bram Wasti
admin_assigned
2024-04-26T07:47:08.369Z
631171f231257261d20f9f57
/avatars/9d6273d9e7dc4e4b21752ba4b17ab1a7.svg
Bram Wasti
user
bwasti
662b1842ef7a4675bdfd9328
Liangzhen Lai
admin_assigned
2024-04-26T07:47:15.337Z
64b705c73240387159397c71
/avatars/da339f07eb58f58756b91eba9af557a1.svg
Liangzhen Lai
user
liangzhen-lai
662b1842ef7a4675bdfd9329
Anas Mahmoud
admin_assigned
2024-04-26T07:47:22.751Z
64a588d26c0223510c6047a9
/avatars/6cc422be5cd2f0f78c0b0cc9c3743e02.svg
Anas Mahmoud
user
nasmahmoud
662b1842ef7a4675bdfd932a
Bilge Acun
admin_assigned
2024-04-26T07:47:30.164Z
650e0307bfb7dd98bba68563
/avatars/1b5287354dc32d85190f38b44ed081f3.svg
Bilge
user
bilgeacun
662b1842ef7a4675bdfd932b
Saurabh Agarwal
662b1842ef7a4675bdfd932c
Ahmed Roman
662b1842ef7a4675bdfd932d
Ahmed A Aly
662b1842ef7a4675bdfd932e
Beidi Chen
admin_assigned
2024-04-26T07:48:13.257Z
64b732f832403871593e082c
/avatars/dd21932b0c167131ee7545a622c46c3c.svg
Beidi Chen
user
beidic
662b1842ef7a4675bdfd932f
Carole-Jean Wu
LayerSkip: Enabling Early Exit Inference and Self-Speculative Decoding
We present LayerSkip, an end-to-end solution to speed-up inference of large
language models (LLMs). First, during training we apply layer dropout, with low
dropout rates for earlier layers and higher dropout rates for later layers, and
an early exit loss where all transformer layers share the same exit. Second,
during inference, we show that this training recipe increases the accuracy of
early exit at earlier layers, without adding any auxiliary layers or modules to
the model. Third, we present a novel self-speculative decoding solution where
we exit at early layers and verify and correct with remaining layers of the
model. Our proposed self-speculative decoding approach has less memory
footprint than other speculative decoding approaches and benefits from shared
compute and activations of the draft and verification stages. We run
experiments on different Llama model sizes on different types of training:
pretraining from scratch, continual pretraining, finetuning on specific data
domain, and finetuning on specific task. We implement our inference solution
and show speedups of up to 2.16x on summarization for CNN/DM documents, 1.82x
on coding, and 2.0x on TOPv2 semantic parsing task. We open source our code and
checkpoints at https://github.com/facebookresearch/LayerSkip.
LayerSkip, an end-to-end method, accelerates large language model inference through layer dropout and self-speculative decoding without auxiliary layers.
layer dropout early exit loss transformer layers early exit self-speculative decoding memory footprint shared compute pretraining continual pretraining finetuning summarization coding semantic parsing