TroL: Traversal of Layers for Large Language and Vision Models
Large language and vision models (LLVMs) have been driven by the
generalization power of large language models (LLMs) and the advent of visual
instruction tuning. Along with scaling them up directly, these models enable
LLVMs to showcase powerful vision language (VL) performances by covering
diverse tasks via natural language instructions. However, existing open-source
LLVMs that perform comparably to closed-source LLVMs such as GPT-4V are often
considered too large (e.g., 26B, 34B, and 110B parameters), having a larger
number of layers. These large models demand costly, high-end resources for both
training and inference. To address this issue, we present a new efficient LLVM
family with 1.8B, 3.8B, and 7B LLM model sizes, Traversal of Layers (TroL),
which enables the reuse of layers in a token-wise manner. This layer traversing
technique simulates the effect of looking back and retracing the answering
stream while increasing the number of forward propagation layers without
physically adding more layers. We demonstrate that TroL employs a simple layer
traversing approach yet efficiently outperforms the open-source LLVMs with
larger model sizes and rivals the performances of the closed-source LLVMs with
substantial sizes.
6672470286f6b3d1267ddce3
Byung-Kwan Lee
claimed_verified
2024-06-19T07:38:10.524Z
657152eb12f162153b50ec9d
https://cdn-avatars.huggingface.co/v1/production/uploads/657152eb12f162153b50ec9d/qnldHP35PclV0pDz_05q8.jpeg
Byung-Kwan Lee
user
BK-Lee
6672470286f6b3d1267ddce4
Sangyun Chung
admin_assigned
2024-06-19T08:09:46.674Z
65a4bf8e90b5e87bcdff41c7
https://cdn-avatars.huggingface.co/v1/production/uploads/65a4bf8e90b5e87bcdff41c7/I6OSoZigV7Fl6OKRl_Yqy.jpeg
Sangyun Chung
user
topyun
6672470286f6b3d1267ddce5
Chae Won Kim
claimed_verified
2024-06-24T07:33:15.975Z
65aa2e84e2a2c86356263340
/avatars/cba9ecceff0f67b2b5dde02e7c0a9991.svg
Chae Won Kim
user
chae-won-kim
6672470286f6b3d1267ddce6
Beomchan Park
claimed_verified
2024-06-19T07:38:08.480Z
65390e7ce279d2fc80a3f800
/avatars/a930c1f20fa1eef372ca92139cb32887.svg
Beomchan Park
user
bpbpbp0810
6672470286f6b3d1267ddce7
Yong Man Ro
admin_assigned
2024-06-19T08:10:04.741Z
660603529e3555d648e3baf1
/avatars/0f54479afcfc19df00b25d5aedb4cf67.svg
Yong Man Ro
user
dwightro
TroL: Traversal of Layers for Large Language and Vision Models
Large language and vision models (LLVMs) have been driven by the
generalization power of large language models (LLMs) and the advent of visual
instruction tuning. Along with scaling them up directly, these models enable
LLVMs to showcase powerful vision language (VL) performances by covering
diverse tasks via natural language instructions. However, existing open-source
LLVMs that perform comparably to closed-source LLVMs such as GPT-4V are often
considered too large (e.g., 26B, 34B, and 110B parameters), having a larger
number of layers. These large models demand costly, high-end resources for both
training and inference. To address this issue, we present a new efficient LLVM
family with 1.8B, 3.8B, and 7B LLM model sizes, Traversal of Layers (TroL),
which enables the reuse of layers in a token-wise manner. This layer traversing
technique simulates the effect of looking back and retracing the answering
stream while increasing the number of forward propagation layers without
physically adding more layers. We demonstrate that TroL employs a simple layer
traversing approach yet efficiently outperforms the open-source LLVMs with
larger model sizes and rivals the performances of the closed-source LLVMs with
substantial sizes.
The TroL model efficiently reduces the size of large language and vision models while maintaining or surpassing performance through a layer traversal technique that reuses layers token-wise.
LLVMs LLMs VL performances visual instruction tuning GPT-4V layer traversing token-wise forward propagation layers