Multimodal Autoregressive Pre-training of Large Vision Encoders
We introduce a novel method for pre-training of large-scale vision encoders.
Building on recent advancements in autoregressive pre-training of vision
models, we extend this framework to a multimodal setting, i.e., images and
text. In this paper, we present AIMV2, a family of generalist vision encoders
characterized by a straightforward pre-training process, scalability, and
remarkable performance across a range of downstream tasks. This is achieved by
pairing the vision encoder with a multimodal decoder that autoregressively
generates raw image patches and text tokens. Our encoders excel not only in
multimodal evaluations but also in vision benchmarks such as localization,
grounding, and classification. Notably, our AIMV2-3B encoder achieves 89.5%
accuracy on ImageNet-1k with a frozen trunk. Furthermore, AIMV2 consistently
outperforms state-of-the-art contrastive models (e.g., CLIP, SigLIP) in
multimodal image understanding across diverse settings.
67400d6480c7c8e332a60bc3
Enrico Fini
claimed_verified
2024-11-22T10:01:48.460Z
639f6ebb9f1f2baab2f69513
/avatars/adb9d4feaf0a6184559b049bfb96ca0d.svg
Enrico Fini
user
efini
67400d6480c7c8e332a60bc4
Mustafa Shukor
admin_assigned
2024-11-22T13:07:33.995Z
62bdeedd01dc22b4d22a371e
/avatars/3cc0643feb53bf2e895ec12c275d5483.svg
Mustafa Shukor
user
mshukor
67400d6480c7c8e332a60bc5
Xiujun Li
admin_assigned
2024-11-22T13:07:42.906Z
6254aa49d21e4cc386b6e6f9
/avatars/ed2ea6dcdf65455cdaaeff4970eddbcd.svg
Xiujun Li
user
xjli
67400d6480c7c8e332a60bc6
Philipp Dufter
admin_assigned
2024-11-22T13:07:48.177Z
6079980b88160e14e4e2e49d
https://cdn-avatars.huggingface.co/v1/production/uploads/6079980b88160e14e4e2e49d/pHROmoIP5xRWNrVufajyf.jpeg
Philipp Dufter
user
pdufter
67400d6480c7c8e332a60bc7
Michal Klein
admin_assigned
2024-11-22T13:07:53.273Z
65a0ff9385f9cc255632c1c6
/avatars/e2884cf99771247408f0a2f0edf58c92.svg
Michal Klein
user
michalk8
67400d6480c7c8e332a60bc8
David Haldimann
67400d6480c7c8e332a60bc9
Sai Aitharaju
67400d6480c7c8e332a60bca
Victor Guilherme Turrisi da Costa
67400d6480c7c8e332a60bcb
Louis Béthune
admin_assigned
2024-11-22T13:08:41.677Z
662a8d7faa7781f8ea521f16
/avatars/c05f9a7885b89c8818d52b98b52ecc7b.svg
Louis Bethune
user
AlgueRythme
67400d6480c7c8e332a60bcc
Zhe Gan
admin_assigned
2024-11-22T13:08:59.126Z
644022db21ec7c165d848b6e
/avatars/d9ed37ec77f2682878b2976a04d2fcd8.svg
Zhe Gan
user
zhegan27
67400d6480c7c8e332a60bcd
Alexander T Toshev
admin_assigned
2024-11-22T13:09:08.756Z
6541636c4f939214d3a0c742
/avatars/c9507d6ec2b81681461837c8261703c7.svg
Alexander Toshev
user
toshev
67400d6480c7c8e332a60bce
Marcin Eichner
67400d6480c7c8e332a60bcf
Moin Nabi
admin_assigned
2024-11-22T13:09:20.776Z
5fe36e9297b72d62ab9b90cd
/avatars/edfea7e8c86dcf355241f5a48365b35a.svg
Moin Nabi
user
moinnabi
67400d6480c7c8e332a60bd0
Yinfei Yang
admin_assigned
2024-11-22T13:09:26.423Z
64b762568c632fbca942a405
/avatars/1eb737ec169967872f1ebf5ff29f1e6b.svg
Yinfei Yang
user
yinfeiy
67400d6480c7c8e332a60bd1
Joshua M. Susskind
admin_assigned
2024-11-22T13:09:32.559Z
6470d2247fd7ecdbd0ec3cc9
/avatars/52c5eca12499a1aa9bd49c43d4f20685.svg
Joshua M. Susskind
user
jsusskind
67400d6480c7c8e332a60bd2
Alaaeldin El-Nouby
Multimodal Autoregressive Pre-training of Large Vision Encoders
We introduce a novel method for pre-training of large-scale vision encoders.
Building on recent advancements in autoregressive pre-training of vision
models, we extend this framework to a multimodal setting, i.e., images and
text. In this paper, we present AIMV2, a family of generalist vision encoders
characterized by a straightforward pre-training process, scalability, and
remarkable performance across a range of downstream tasks. This is achieved by
pairing the vision encoder with a multimodal decoder that autoregressively
generates raw image patches and text tokens. Our encoders excel not only in
multimodal evaluations but also in vision benchmarks such as localization,
grounding, and classification. Notably, our AIMV2-3B encoder achieves 89.5%
accuracy on ImageNet-1k with a frozen trunk. Furthermore, AIMV2 consistently
outperforms state-of-the-art contrastive models (e.g., CLIP, SigLIP) in
multimodal image understanding across diverse settings.
AIMV2, a multimodal vision encoder paired with an autoregressive decoder, achieves superior performance in vision benchmarks and multimodal image understanding tasks compared to contrastive models.
autoregressive pre-training vision encoders multimodal setting multimodal decoder raw image patches text tokens localization grounding classification ImageNet-1k frozen trunk contrastive models CLIP SigLIP