AlignGPT: Multi-modal Large Language Models with Adaptive Alignment
  Capability
Multimodal Large Language Models (MLLMs) are widely regarded as crucial in
the exploration of Artificial General Intelligence (AGI). The core of MLLMs
lies in their capability to achieve cross-modal alignment. To attain this goal,
current MLLMs typically follow a two-phase training paradigm: the pre-training
phase and the instruction-tuning phase. Despite their success, there are
shortcomings in the modeling of alignment capabilities within these models.
Firstly, during the pre-training phase, the model usually assumes that all
image-text pairs are uniformly aligned, but in fact the degree of alignment
between different image-text pairs is inconsistent. Secondly, the instructions
currently used for finetuning incorporate a variety of tasks, different tasks's
instructions usually require different levels of alignment capabilities, but
previous MLLMs overlook these differentiated alignment needs. To tackle these
issues, we propose a new multimodal large language model AlignGPT. In the
pre-training stage, instead of treating all image-text pairs equally, we assign
different levels of alignment capabilities to different image-text pairs. Then,
in the instruction-tuning phase, we adaptively combine these different levels
of alignment capabilities to meet the dynamic alignment needs of different
instructions. Extensive experimental results show that our model achieves
competitive performance on 12 benchmarks.
66501fe060fc035edd879d5a
Fei Zhao
claimed_verified
2024-05-24T15:44:38.309Z
643660b5fae28700514a138d
/avatars/a48bddbfdcedc0dd2c2836202b3c8207.svg
zhaofei
user
nlpzhaof
66501fe060fc035edd879d5b
Taotian Pang
admin_assigned
2024-05-24T11:57:30.613Z
64b10448d47ab770167865e4
/avatars/d56a3210f1460c4c46fe782eee9cb8c0.svg
Taotian Pang
user
TJTP
66501fe060fc035edd879d5c
Chunhui Li
66501fe060fc035edd879d5d
Zhen Wu
66501fe060fc035edd879d5e
Junjie Guo
admin_assigned
2024-05-24T11:59:16.605Z
639598cb2caa32c0d334a545
/avatars/2751ce681232b0b4666f87fd3f20aebd.svg
Junjie Guo
user
TealOrange
66501fe060fc035edd879d5f
Shangyu Xing
admin_assigned
2024-05-24T11:59:22.829Z
64bc8f584d2052b1aa062b91
/avatars/c63f3e4787f463d36bcdf8c15d8214c1.svg
Shangyu Xing
user
starreeze
66501fe060fc035edd879d60
Xinyu Dai
AlignGPT: Multi-modal Large Language Models with Adaptive Alignment
  Capability
Multimodal Large Language Models (MLLMs) are widely regarded as crucial in
the exploration of Artificial General Intelligence (AGI). The core of MLLMs
lies in their capability to achieve cross-modal alignment. To attain this goal,
current MLLMs typically follow a two-phase training paradigm: the pre-training
phase and the instruction-tuning phase. Despite their success, there are
shortcomings in the modeling of alignment capabilities within these models.
Firstly, during the pre-training phase, the model usually assumes that all
image-text pairs are uniformly aligned, but in fact the degree of alignment
between different image-text pairs is inconsistent. Secondly, the instructions
currently used for finetuning incorporate a variety of tasks, different tasks's
instructions usually require different levels of alignment capabilities, but
previous MLLMs overlook these differentiated alignment needs. To tackle these
issues, we propose a new multimodal large language model AlignGPT. In the
pre-training stage, instead of treating all image-text pairs equally, we assign
different levels of alignment capabilities to different image-text pairs. Then,
in the instruction-tuning phase, we adaptively combine these different levels
of alignment capabilities to meet the dynamic alignment needs of different
instructions. Extensive experimental results show that our model achieves
competitive performance on 12 benchmarks.
AlignGPT addresses alignment inconsistencies in multimodal large language models by dynamically assigning and combining varying alignment capabilities during pre-training and instruction tuning phases.
Multimodal Large Language Models AGI cross-modal alignment pre-training phase instruction-tuning phase image-text pairs AlignGPT