Router-Tuning: A Simple and Effective Approach for Enabling
  Dynamic-Depth in Transformers
Traditional transformer models often allocate a fixed amount of computational
resources to every input token, leading to inefficient and unnecessary
computation. To address this, the Mixture of Depths (MoD) was introduced to
dynamically adjust the computational depth by skipping less important layers.
Despite its promise, current MoD approaches remain under-explored and face two
main challenges: (1) high training costs due to the need to train the
entire model along with the routers that determine which layers to skip, and
(2) the risk of performance degradation when important layers are
bypassed. In response to the first issue, we propose Router-Tuning, a method
that fine-tunes only the router on a small dataset, drastically reducing the
computational overhead associated with full model training. For the second
challenge, we propose MindSkip, which deploys Attention with Dynamic
Depths. This method preserves the model's performance while significantly
enhancing computational and memory efficiency. Extensive experiments
demonstrate that our approach delivers competitive results while dramatically
improving the computation efficiency, e.g., 21\% speedup and only a 0.2\%
performance drop. The code is released at
https://github.com/CASE-Lab-UMD/Router-Tuning.
6717ac546136223abbfd80bf
Shwai He
admin_assigned
2024-10-22T13:48:00.289Z
64468d7a3b28db22e7f84418
/avatars/ff05994acd7da64a2ed6e44f12cf77bc.svg
Shwai He
user
Shwai
6717ac546136223abbfd80c0
Tao Ge
6717ac546136223abbfd80c1
Guoheng Sun
admin_assigned
2024-10-22T13:48:19.181Z
6438f5555aa69077ffb16218
/avatars/582f736c0485252010b682e85db62e77.svg
Guoheng Sun
user
s1ghhh
6717ac546136223abbfd80c2
Bowei Tian
admin_assigned
2024-10-22T13:48:27.178Z
6623e96dd7b6076f66480898
/avatars/62288faa8ecc69c6b1cef4bc553941b9.svg
Bowei Tian
user
abdd68
6717ac546136223abbfd80c3
Xiaoyang Wang
claimed_verified
2024-10-30T09:55:19.525Z
657cd228138b7e391444a65d
/avatars/c7c984ae483144fab627aa2c54d91d0f.svg
Xiaoyang Wang
user
xywang1
6717ac546136223abbfd80c4
Ang Li
6717ac546136223abbfd80c5
Dong Yu
Router-Tuning: A Simple and Effective Approach for Enabling
  Dynamic-Depth in Transformers
Traditional transformer models often allocate a fixed amount of computational
resources to every input token, leading to inefficient and unnecessary
computation. To address this, the Mixture of Depths (MoD) was introduced to
dynamically adjust the computational depth by skipping less important layers.
Despite its promise, current MoD approaches remain under-explored and face two
main challenges: (1) high training costs due to the need to train the
entire model along with the routers that determine which layers to skip, and
(2) the risk of performance degradation when important layers are
bypassed. In response to the first issue, we propose Router-Tuning, a method
that fine-tunes only the router on a small dataset, drastically reducing the
computational overhead associated with full model training. For the second
challenge, we propose MindSkip, which deploys Attention with Dynamic
Depths. This method preserves the model's performance while significantly
enhancing computational and memory efficiency. Extensive experiments
demonstrate that our approach delivers competitive results while dramatically
improving the computation efficiency, e.g., 21\% speedup and only a 0.2\%
performance drop. The code is released at
https://github.com/CASE-Lab-UMD/Router-Tuning.
Router-Tuning and MindSkip improve the efficiency of Mixture of Depths by selectively fine-tuning the router and using dynamic depths with attention, respectively.
Mixture of Depths (MoD) Router-Tuning MindSkip Attention with Dynamic Depths