EE-LLM: Large-Scale Training and Inference of Early-Exit Large Language
  Models with 3D Parallelism
We present EE-LLM, a framework for large-scale training and inference of
early-exit large language models (LLMs). While recent works have shown
preliminary evidence for the efficacy of early exiting in accelerating LLM
inference, EE-LLM makes a foundational step towards scaling up early-exit LLMs
by supporting their training and inference with massive 3D parallelism. Built
upon Megatron-LM, EE-LLM implements a variety of algorithmic innovations and
performance optimizations tailored to early exiting, including a lightweight
method that facilitates backpropagation for the early-exit training objective
with pipeline parallelism, techniques of leveraging idle resources in the
original pipeline schedule for computation related to early-exit layers, and
two approaches of early-exit inference that are compatible with KV caching for
autoregressive generation. Our analytical and empirical study shows that EE-LLM
achieves great training efficiency with negligible computational overhead
compared to standard LLM training, as well as outstanding inference speedup
without compromising output quality. To facilitate further research and
adoption, we release EE-LLM at https://github.com/pan-x-c/EE-LLM.
6576763db238c76bba229aab
Yanxi Chen
claimed_verified
2023-12-11T12:37:11.761Z
6576f9f4654561a1b345610b
/avatars/f801f551640caa70368fcc26a0f51d27.svg
Yanxi Chen
user
yanxi-chen
6576763db238c76bba229aac
Xuchen Pan
admin_assigned
2023-12-11T08:45:56.504Z
64101b4ca92fedb0e8559b91
/avatars/c7453a5641a94d04a75de9b7d4272bc2.svg
Xuchen Pan
user
panxuchen
6576763db238c76bba229aad
Yaliang Li
6576763db238c76bba229aae
Bolin Ding
6576763db238c76bba229aaf
Jingren Zhou
admin_assigned
2023-12-11T08:46:23.863Z
602f88f5e8149a962412a667
/avatars/b78f0e583df8e5d5e3365934fe5f4900.svg
Zhou
user
Jingren
EE-LLM: Large-Scale Training and Inference of Early-Exit Large Language
  Models with 3D Parallelism
We present EE-LLM, a framework for large-scale training and inference of
early-exit large language models (LLMs). While recent works have shown
preliminary evidence for the efficacy of early exiting in accelerating LLM
inference, EE-LLM makes a foundational step towards scaling up early-exit LLMs
by supporting their training and inference with massive 3D parallelism. Built
upon Megatron-LM, EE-LLM implements a variety of algorithmic innovations and
performance optimizations tailored to early exiting, including a lightweight
method that facilitates backpropagation for the early-exit training objective
with pipeline parallelism, techniques of leveraging idle resources in the
original pipeline schedule for computation related to early-exit layers, and
two approaches of early-exit inference that are compatible with KV caching for
autoregressive generation. Our analytical and empirical study shows that EE-LLM
achieves great training efficiency with negligible computational overhead
compared to standard LLM training, as well as outstanding inference speedup
without compromising output quality. To facilitate further research and
adoption, we release EE-LLM at https://github.com/pan-x-c/EE-LLM.
EE-LLM is a framework that enhances early-exit large language model training and inference using 3D parallelism and performance optimizations.
early-exit large language models 3D parallelism Megatron-LM pipeline parallelism idle resources autoregressive generation KV caching