MM1: Methods, Analysis & Insights from Multimodal LLM Pre-training
In this work, we discuss building performant Multimodal Large Language Models
(MLLMs). In particular, we study the importance of various architecture
components and data choices. Through careful and comprehensive ablations of the
image encoder, the vision language connector, and various pre-training data
choices, we identified several crucial design lessons. For example, we
demonstrate that for large-scale multimodal pre-training using a careful mix of
image-caption, interleaved image-text, and text-only data is crucial for
achieving state-of-the-art (SOTA) few-shot results across multiple benchmarks,
compared to other published pre-training results. Further, we show that the
image encoder together with image resolution and the image token count has
substantial impact, while the vision-language connector design is of
comparatively negligible importance. By scaling up the presented recipe, we
build MM1, a family of multimodal models up to 30B parameters, consisting of
both dense models and mixture-of-experts (MoE) variants, that are SOTA in
pre-training metrics and achieve competitive performance after supervised
fine-tuning on a range of established multimodal benchmarks. Thanks to
large-scale pre-training, MM1 enjoys appealing properties such as enhanced
in-context learning, and multi-image reasoning, enabling few-shot
chain-of-thought prompting.
65f3b1fb7c44e9e26df10330
Brandon McKinzie
extracted_confirmed
2024-03-15T14:48:40.375Z
65a16b5bfb3898fe81c63f51
/avatars/828e8f3a8efe8c5eb07eb2c0796809d2.svg
Brandon McKinzie
user
bmckinz
65f3b1fb7c44e9e26df10331
Zhe Gan
65f3b1fb7c44e9e26df10332
Jean-Philippe Fauconnier
claimed_verified
2024-07-02T21:09:15.876Z
62cd5c43299c0c2e0e437842
/avatars/8fa8511baf2d9bd95d3ba4535a5b3d69.svg
Jean-Philippe Fauconnier
user
j4kn
65f3b1fb7c44e9e26df10333
Sam Dodge
65f3b1fb7c44e9e26df10334
Bowen Zhang
65f3b1fb7c44e9e26df10335
Philipp Dufter
admin_assigned
2024-03-15T08:13:01.977Z
6079980b88160e14e4e2e49d
https://cdn-avatars.huggingface.co/v1/production/uploads/6079980b88160e14e4e2e49d/pHROmoIP5xRWNrVufajyf.jpeg
Philipp Dufter
user
pdufter
65f3b1fb7c44e9e26df10336
Dhruti Shah
admin_assigned
2024-03-15T08:13:14.745Z
65a686f7f319aa03481ed105
/avatars/0e344f4ff0714932340b6225eadfcd6d.svg
Dhruti Shah
user
dhruti96shah
65f3b1fb7c44e9e26df10337
Xianzhi Du
admin_assigned
2024-03-15T08:13:23.713Z
65cc30b80390fce6291d03cf
/avatars/7e774270d1cca48f43f0b379af87003e.svg
Xianzhi Du
user
xianzhi-du
65f3b1fb7c44e9e26df10338
Futang Peng
65f3b1fb7c44e9e26df10339
Floris Weers
claimed_verified
2024-03-15T11:02:56.703Z
65f41f1d6724ea19d7bc9052
/avatars/3171a092b40da7518bce00b80f7c6ff0.svg
Floris Weers
user
fweers
65f3b1fb7c44e9e26df1033a
Anton Belyi
admin_assigned
2024-03-15T08:13:45.933Z
656b630a9c8778992f7966a4
/avatars/6438033af2f1abc3b6761cd5cac92cb8.svg
Anton Belyi
user
antonis-white
65f3b1fb7c44e9e26df1033b
Haotian Zhang
claimed_verified
2024-03-15T08:03:10.698Z
631516348d85ad332fa47b2c
/avatars/100f5ae3cf3c52faaecdaecd5d8f2881.svg
Haotian Zhang
user
haotiz
65f3b1fb7c44e9e26df1033c
Karanjeet Singh
admin_assigned
2024-03-15T08:13:55.064Z
649b97d86104e208ef64bb6f
/avatars/f93682073b4a2dd67ca223439294ed56.svg
karanjeet singh
user
sorryihaveagf
65f3b1fb7c44e9e26df1033d
Doug Kang
65f3b1fb7c44e9e26df1033e
Hongyu Hè
65f3b1fb7c44e9e26df1033f
Max Schwarzer
admin_assigned
2024-03-15T08:14:21.670Z
6306f2de435ec751b730087e
/avatars/7e2858cbc1526f6060a730f8ca3af290.svg
Max Schwarzer
user
MaxSchwarzer
65f3b1fb7c44e9e26df10340
Tom Gunter
65f3b1fb7c44e9e26df10341
Xiang Kong
65f3b1fb7c44e9e26df10342
Aonan Zhang
claimed_verified
2024-03-16T10:37:28.641Z
65f3e68a138c6ab771434e2d
/avatars/7bfbdb1949f73b3d8f88ae2ff73900bb.svg
Aonan Zhang
user
AonanZhang
65f3b1fb7c44e9e26df10343
Jianyu Wang
65f3b1fb7c44e9e26df10344
Chong Wang
65f3b1fb7c44e9e26df10345
Nan Du
65f3b1fb7c44e9e26df10346
Tao Lei
65f3b1fb7c44e9e26df10347
Sam Wiseman
admin_assigned
2024-03-15T08:16:00.960Z
64a435475db6c484bc9a9bd4
/avatars/f5ecc5affd2d5064fd21b50e0ea0fa0e.svg
Sam Wiseman
user
swiseman
65f3b1fb7c44e9e26df10357
Guoli Yin
claimed_verified
2024-07-30T09:51:05.703Z
605b7f42935268bc086131ba
/avatars/a55109f714b33f9d59d69011ddeb0b9f.svg
Guoli Yin
user
gyin94
65f3b1fb7c44e9e26df10348
Mark Lee
65f3b1fb7c44e9e26df10349
Zirui Wang
65f3b1fb7c44e9e26df1034a
Ruoming Pang
admin_assigned
2024-03-15T08:16:56.862Z
654ef8ad3fe6c0b1f871942f
/avatars/8d3689b9bf57c7c8060ac510a1839158.svg
Ruoming Pang
user
ruoming
65f3b1fb7c44e9e26df1034b
Peter Grasch
65f3b1fb7c44e9e26df1034c
Alexander Toshev
admin_assigned
2024-03-15T08:16:44.937Z
6541636c4f939214d3a0c742
/avatars/c9507d6ec2b81681461837c8261703c7.svg
Alexander Toshev
user
toshev
65f3b1fb7c44e9e26df1034d
Yinfei Yang
admin_assigned
2024-03-15T08:16:36.271Z
64b762568c632fbca942a405
/avatars/1eb737ec169967872f1ebf5ff29f1e6b.svg
Yinfei Yang
user
yinfeiy
MM1: Methods, Analysis & Insights from Multimodal LLM Pre-training
In this work, we discuss building performant Multimodal Large Language Models
(MLLMs). In particular, we study the importance of various architecture
components and data choices. Through careful and comprehensive ablations of the
image encoder, the vision language connector, and various pre-training data
choices, we identified several crucial design lessons. For example, we
demonstrate that for large-scale multimodal pre-training using a careful mix of
image-caption, interleaved image-text, and text-only data is crucial for
achieving state-of-the-art (SOTA) few-shot results across multiple benchmarks,
compared to other published pre-training results. Further, we show that the
image encoder together with image resolution and the image token count has
substantial impact, while the vision-language connector design is of
comparatively negligible importance. By scaling up the presented recipe, we
build MM1, a family of multimodal models up to 30B parameters, consisting of
both dense models and mixture-of-experts (MoE) variants, that are SOTA in
pre-training metrics and achieve competitive performance after supervised
fine-tuning on a range of established multimodal benchmarks. Thanks to
large-scale pre-training, MM1 enjoys appealing properties such as enhanced
in-context learning, and multi-image reasoning, enabling few-shot
chain-of-thought prompting.
A study of Multimodal Large Language Models finds a mix of pre-training data and image encoder design crucial for SOTA performance on multimodal benchmarks.
Multimodal Large Language Models image encoder vision language connector pre-training data image-caption interleaved image-text text-only mixture-of-experts few-shot results in-context learning multi-image reasoning chain-of-thought prompting