Adapting LLaMA Decoder to Vision Transformer
This work examines whether decoder-only Transformers such as LLaMA, which
were originally designed for large language models (LLMs), can be adapted to
the computer vision field. We first "LLaMAfy" a standard ViT step-by-step to
align with LLaMA's architecture, and find that directly applying a casual mask
to the self-attention brings an attention collapse issue, resulting in the
failure to the network training. We suggest to reposition the class token
behind the image tokens with a post-sequence class token technique to overcome
this challenge, enabling causal self-attention to efficiently capture the
entire image's information. Additionally, we develop a soft mask strategy that
gradually introduces a casual mask to the self-attention at the onset of
training to facilitate the optimization behavior. The tailored model, dubbed as
image LLaMA (iLLaMA), is akin to LLaMA in architecture and enables direct
supervised learning. Its causal self-attention boosts computational efficiency
and learns complex representation by elevating attention map ranks. iLLaMA
rivals the performance with its encoder-only counterparts, achieving 75.1%
ImageNet top-1 accuracy with only 5.7M parameters. Scaling the model to ~310M
and pre-training on ImageNet-21K further enhances the accuracy to 86.0%.
Extensive experiments demonstrate iLLaMA's reliable properties: calibration,
shape-texture bias, quantization compatibility, ADE20K segmentation and CIFAR
transfer learning. We hope our study can kindle fresh views to visual model
design in the wave of LLMs. Pre-trained models and codes are available here.
66173f994488ff8dfa933e33
Jiahao Wang
claimed_verified
2024-04-11T09:02:32.979Z
65994517d3f2137415bb4d5b
https://cdn-avatars.huggingface.co/v1/production/uploads/65994517d3f2137415bb4d5b/PhjWetLTMn3bThWFLGCxR.png
WANG Jiahao
user
techmonsterwang
66173f994488ff8dfa933e34
Wenqi Shao
admin_assigned
2024-04-18T14:18:52.599Z
64b3fd42eec33e27dcc4c941
/avatars/5aa1a99468fa61d4b8b0e80b592c4e55.svg
Wenqi Shao
user
wqshao126
66173f994488ff8dfa933e35
Mengzhao Chen
admin_assigned
2024-04-18T14:19:09.508Z
64aea082704210bf815e7551
/avatars/5c8dc0df57596c526b2bccea21835f53.svg
Mengzhao Chen
user
ChenMnZ
66173f994488ff8dfa933e36
Chengyue Wu
admin_assigned
2024-04-18T14:19:24.943Z
617526c9de8feb54b0ce45ad
/avatars/7faf8c6f71fc318a0113d780d376c381.svg
Wu Chengyue
user
WuChengyue
66173f994488ff8dfa933e37
Yong Liu
66173f994488ff8dfa933e38
Kaipeng Zhang
admin_assigned
2024-04-18T14:19:37.309Z
63527f4e7d071f23d085ad45
/avatars/99a51adef5673b3ac1a8c02eb47759c4.svg
KAIPENG ZHANG
user
kpzhang
66173f994488ff8dfa933e39
Songyang Zhang
admin_assigned
2024-04-18T14:20:17.538Z
630716d11801ecc7d2595021
/avatars/2d36a880ce4a3cf7efc5ff3987dbeaf3.svg
Songyang Zhang
user
zsytony
66173f994488ff8dfa933e3a
Kai Chen
66173f994488ff8dfa933e3b
Ping Luo
Adapting LLaMA Decoder to Vision Transformer
This work examines whether decoder-only Transformers such as LLaMA, which
were originally designed for large language models (LLMs), can be adapted to
the computer vision field. We first "LLaMAfy" a standard ViT step-by-step to
align with LLaMA's architecture, and find that directly applying a casual mask
to the self-attention brings an attention collapse issue, resulting in the
failure to the network training. We suggest to reposition the class token
behind the image tokens with a post-sequence class token technique to overcome
this challenge, enabling causal self-attention to efficiently capture the
entire image's information. Additionally, we develop a soft mask strategy that
gradually introduces a casual mask to the self-attention at the onset of
training to facilitate the optimization behavior. The tailored model, dubbed as
image LLaMA (iLLaMA), is akin to LLaMA in architecture and enables direct
supervised learning. Its causal self-attention boosts computational efficiency
and learns complex representation by elevating attention map ranks. iLLaMA
rivals the performance with its encoder-only counterparts, achieving 75.1%
ImageNet top-1 accuracy with only 5.7M parameters. Scaling the model to ~310M
and pre-training on ImageNet-21K further enhances the accuracy to 86.0%.
Extensive experiments demonstrate iLLaMA's reliable properties: calibration,
shape-texture bias, quantization compatibility, ADE20K segmentation and CIFAR
transfer learning. We hope our study can kindle fresh views to visual model
design in the wave of LLMs. Pre-trained models and codes are available here.
A decoder-only Transformer model, LLaMA, is adapted for computer vision tasks through architectural modifications, achieving performance comparable to encoder-only models with fewer parameters and improved attention mechanisms.
decoder-only Transformers LLaMA large language models LLMs ViT causal mask self-attention attention collapse class token post-sequence class token soft mask image LLaMA iLLaMA computational efficiency attention map ranks ImageNet pre-training ImageNet-21K calibration shape-texture bias quantization compatibility ADE20K segmentation CIFAR transfer learning