MM1.5: Methods, Analysis & Insights from Multimodal LLM Fine-tuning
We present MM1.5, a new family of multimodal large language models (MLLMs)
designed to enhance capabilities in text-rich image understanding, visual
referring and grounding, and multi-image reasoning. Building upon the MM1
architecture, MM1.5 adopts a data-centric approach to model training,
systematically exploring the impact of diverse data mixtures across the entire
model training lifecycle. This includes high-quality OCR data and synthetic
captions for continual pre-training, as well as an optimized visual
instruction-tuning data mixture for supervised fine-tuning. Our models range
from 1B to 30B parameters, encompassing both dense and mixture-of-experts (MoE)
variants, and demonstrate that careful data curation and training strategies
can yield strong performance even at small scales (1B and 3B). Additionally, we
introduce two specialized variants: MM1.5-Video, designed for video
understanding, and MM1.5-UI, tailored for mobile UI understanding. Through
extensive empirical studies and ablations, we provide detailed insights into
the training processes and decisions that inform our final designs, offering
valuable guidance for future research in MLLM development.
66fb8eda0c0759cfa0920c46
Haotian Zhang
claimed_verified
2024-10-01T07:16:34.961Z
631516348d85ad332fa47b2c
/avatars/100f5ae3cf3c52faaecdaecd5d8f2881.svg
Haotian Zhang
user
haotiz
66fb8eda0c0759cfa0920c47
Mingfei Gao
admin_assigned
2024-10-01T07:34:18.731Z
64388d5f1efe72ba48071c52
/avatars/2653fba13e71f471e9d55c134fe25efc.svg
Mingfei Gao
user
fly6464
66fb8eda0c0759cfa0920c48
Zhe Gan
admin_assigned
2024-10-01T07:34:38.598Z
644022db21ec7c165d848b6e
/avatars/d9ed37ec77f2682878b2976a04d2fcd8.svg
Zhe Gan
user
zhegan27
66fb8eda0c0759cfa0920c49
Philipp Dufter
admin_assigned
2024-10-01T07:34:44.643Z
6079980b88160e14e4e2e49d
https://cdn-avatars.huggingface.co/v1/production/uploads/6079980b88160e14e4e2e49d/pHROmoIP5xRWNrVufajyf.jpeg
Philipp Dufter
user
pdufter
66fb8eda0c0759cfa0920c4a
Nina Wenzel
admin_assigned
2024-10-01T07:34:50.560Z
65f9a3ebf8c8ab9966fccffc
/avatars/07e466225e1142d2fd7ebc2f23fe5062.svg
Nina Wenzel
user
nm-w
66fb8eda0c0759cfa0920c4b
Forrest Huang
66fb8eda0c0759cfa0920c4c
Dhruti Shah
admin_assigned
2024-10-01T07:35:11.193Z
65a686f7f319aa03481ed105
/avatars/0e344f4ff0714932340b6225eadfcd6d.svg
Dhruti Shah
user
dhruti96shah
66fb8eda0c0759cfa0920c4d
Xianzhi Du
admin_assigned
2024-10-01T07:35:17.781Z
65cc30b80390fce6291d03cf
/avatars/7e774270d1cca48f43f0b379af87003e.svg
Xianzhi Du
user
xianzhi-du
66fb8eda0c0759cfa0920c4e
Bowen Zhang
66fb8eda0c0759cfa0920c4f
Yanghao Li
admin_assigned
2024-10-01T07:35:45.486Z
65dad3870af7e21ba473439f
/avatars/da542e7d68ae937bbdb791f17096bb1c.svg
Yanghao Li
user
FrozzZen
66fb8eda0c0759cfa0920c50
Sam Dodge
66fb8eda0c0759cfa0920c51
Keen You
admin_assigned
2024-10-01T07:35:59.099Z
6150a95100a52e55be2994eb
/avatars/c8e2f85a00ca54e1984a692f29f3096f.svg
Keen You
user
keenyou
66fb8eda0c0759cfa0920c52
Zhen Yang
claimed_verified
2024-10-02T07:41:24.739Z
66047f6f0e127d15dcfc262f
/avatars/a2355f30a358fed4ae6bf600ec4be9c7.svg
Zhen Yang
user
zhen-yang
66fb8eda0c0759cfa0920c53
Aleksei Timofeev
admin_assigned
2024-10-01T07:36:26.873Z
6473ae08485a7c8e1bd34e0d
/avatars/575faf4b3e1dc7b657ed18fac9b9501a.svg
Aleksei Timofeev
user
Ymaril
66fb8eda0c0759cfa0920c54
Mingze Xu
admin_assigned
2024-10-01T07:36:51.729Z
65fc655930aee594d74dd7b8
/avatars/e8df7170fa1cf99bb3a2b9a083ee99a1.svg
Mingze Xu
user
xumingze0308
66fb8eda0c0759cfa0920c55
Hong-You Chen
admin_assigned
2024-10-01T07:36:57.955Z
64b720891a4d97b5d7f055b8
/avatars/7247a4f3a433a58fe934f245d9a757b2.svg
Hong-You Chen
user
Cbbjames
66fb8eda0c0759cfa0920c56
Jean-Philippe Fauconnier
admin_assigned
2024-10-01T07:37:04.189Z
62cd5c43299c0c2e0e437842
/avatars/8fa8511baf2d9bd95d3ba4535a5b3d69.svg
Jean-Philippe Fauconnier
user
j4kn
66fb8eda0c0759cfa0920c57
Zhengfeng Lai
claimed_verified
2024-10-01T07:34:23.314Z
66b5295f83425904fa7a1a6a
/avatars/a35568fb933ceef7451bd88fb3d5ab17.svg
Zhengfeng Lai
user
jefflai
66fb8eda0c0759cfa0920c58
Haoxuan You
admin_assigned
2024-10-01T07:37:20.593Z
65284ba3e3b3844f5e487a0b
/avatars/fce12a233bddb611d441f25a2ab17725.svg
Haoxuan You
user
HaoxuanYou
66fb8eda0c0759cfa0920c59
Zirui Wang
66fb8eda0c0759cfa0920c5a
Afshin Dehghan
66fb8eda0c0759cfa0920c5b
Peter Grasch
admin_assigned
2024-10-01T07:38:00.608Z
6649e0b6f50d471119285ab8
/avatars/1c93d0e15f2b68281a1a2ea052450a45.svg
Peter Grasch
user
petergrasch
66fb8eda0c0759cfa0920c5c
Yinfei Yang
admin_assigned
2024-10-01T07:38:07.212Z
64b762568c632fbca942a405
/avatars/1eb737ec169967872f1ebf5ff29f1e6b.svg
Yinfei Yang
user
yinfeiy
MM1.5: Methods, Analysis & Insights from Multimodal LLM Fine-tuning
We present MM1.5, a new family of multimodal large language models (MLLMs)
designed to enhance capabilities in text-rich image understanding, visual
referring and grounding, and multi-image reasoning. Building upon the MM1
architecture, MM1.5 adopts a data-centric approach to model training,
systematically exploring the impact of diverse data mixtures across the entire
model training lifecycle. This includes high-quality OCR data and synthetic
captions for continual pre-training, as well as an optimized visual
instruction-tuning data mixture for supervised fine-tuning. Our models range
from 1B to 30B parameters, encompassing both dense and mixture-of-experts (MoE)
variants, and demonstrate that careful data curation and training strategies
can yield strong performance even at small scales (1B and 3B). Additionally, we
introduce two specialized variants: MM1.5-Video, designed for video
understanding, and MM1.5-UI, tailored for mobile UI understanding. Through
extensive empirical studies and ablations, we provide detailed insights into
the training processes and decisions that inform our final designs, offering
valuable guidance for future research in MLLM development.
MM1.5 enhances multimodal language models with a data-centric approach, supporting diverse tasks like text-rich image understanding and video comprehension through careful data curation and training strategies.
multimodal large language models MLLMs data-centric approach high-quality OCR data synthetic captions visual instruction-tuning dense mixture-of-experts MoE MM1.5-Video MM1.5-UI video understanding mobile UI understanding empirical studies ablations