The Synergy between Data and Multi-Modal Large Language Models: A Survey
  from Co-Development Perspective
The rapid development of large language models (LLMs) has been witnessed in
recent years. Based on the powerful LLMs, multi-modal LLMs (MLLMs) extend the
modality from text to a broader spectrum of domains, attracting widespread
attention due to the broader range of application scenarios. As LLMs and MLLMs
rely on vast amounts of model parameters and data to achieve emergent
capabilities, the importance of data is receiving increasingly widespread
attention and recognition. Tracing and analyzing recent data-oriented works for
MLLMs, we find that the development of models and data is not two separate
paths but rather interconnected. On the one hand, vaster and higher-quality
data contribute to better performance of MLLMs, on the other hand, MLLMs can
facilitate the development of data. The co-development of multi-modal data and
MLLMs requires a clear view of 1) at which development stage of MLLMs can
specific data-centric approaches be employed to enhance which capabilities, and
2) by utilizing which capabilities and acting as which roles can models
contribute to multi-modal data. To promote the data-model co-development for
MLLM community, we systematically review existing works related to MLLMs from
the data-model co-development perspective. A regularly maintained project
associated with this survey is accessible at
https://github.com/modelscope/data-juicer/blob/main/docs/awesome_llm_data.md.
66908e4644464c66a819e74e
Zhen Qin
claimed_verified
2024-07-12T07:40:14.493Z
649014b91d71e55664838d2d
/avatars/f0e0f2830c5cb7428cbbc9634d95c34b.svg
Zhen Qin
user
zhenqincn
66908e4644464c66a819e74f
Daoyuan Chen
admin_assigned
2024-07-12T10:09:23.541Z
6530ca70c1b8f265a431fc42
/avatars/7b1ec5b75398c8434ccd43cdce7fee53.svg
Daoyuan Chen
user
yxdyc
66908e4644464c66a819e750
Wenhao Zhang
66908e4644464c66a819e751
Liuyi Yao
admin_assigned
2024-07-12T10:09:39.519Z
649148b63c7b190918fc9214
/avatars/1dc51c948cd8e4ab716a8b199ef04387.svg
Liuyi Yao
user
Briayao
66908e4644464c66a819e752
Yilun Huang
claimed_verified
2025-07-07T11:27:41.982Z
64213e9d9628d83384af2632
/avatars/46a34fe25b139acf59f8e5a665999ba1.svg
Dylan Huang
user
hylcool
66908e4644464c66a819e753
Bolin Ding
66908e4644464c66a819e754
Yaliang Li
66908e4644464c66a819e755
Shuiguang Deng
The Synergy between Data and Multi-Modal Large Language Models: A Survey
  from Co-Development Perspective
The rapid development of large language models (LLMs) has been witnessed in
recent years. Based on the powerful LLMs, multi-modal LLMs (MLLMs) extend the
modality from text to a broader spectrum of domains, attracting widespread
attention due to the broader range of application scenarios. As LLMs and MLLMs
rely on vast amounts of model parameters and data to achieve emergent
capabilities, the importance of data is receiving increasingly widespread
attention and recognition. Tracing and analyzing recent data-oriented works for
MLLMs, we find that the development of models and data is not two separate
paths but rather interconnected. On the one hand, vaster and higher-quality
data contribute to better performance of MLLMs, on the other hand, MLLMs can
facilitate the development of data. The co-development of multi-modal data and
MLLMs requires a clear view of 1) at which development stage of MLLMs can
specific data-centric approaches be employed to enhance which capabilities, and
2) by utilizing which capabilities and acting as which roles can models
contribute to multi-modal data. To promote the data-model co-development for
MLLM community, we systematically review existing works related to MLLMs from
the data-model co-development perspective. A regularly maintained project
associated with this survey is accessible at
https://github.com/modelscope/data-juicer/blob/main/docs/awesome_llm_data.md.
A review of the co-development between multi-modal language models and data highlights the interdependence between model advancements and data quality.
large language models multi-modal language models model parameters vaster and higher-quality data co-development data-centric approaches multi-modal data