u-LLaVA: Unifying Multi-Modal Tasks via Large Language Model
Recent advances such as LLaVA and Mini-GPT4 have successfully integrated
visual information into LLMs, yielding inspiring outcomes and giving rise to a
new generation of multi-modal LLMs, or MLLMs. Nevertheless, these methods
struggle with hallucinations and the mutual interference between tasks. To
tackle these problems, we propose an efficient and accurate approach to adapt
to downstream tasks by utilizing LLM as a bridge to connect multiple expert
models, namely u-LLaVA. Firstly, we incorporate the modality alignment module
and multi-task modules into LLM. Then, we reorganize or rebuild multi-type
public datasets to enable efficient modality alignment and instruction
following. Finally, task-specific information is extracted from the trained LLM
and provided to different modules for solving downstream tasks. The overall
framework is simple, effective, and achieves state-of-the-art performance
across multiple benchmarks. We also release our model, the generated data, and
the code base publicly available.
654db51d0011607c312089c7
Jinjin Xu
654db51d0011607c312089c8
Liwu Xu
admin_assigned
2023-11-10T11:14:18.355Z
641f47743c49fc33ced8f414
/avatars/e8269d87541f3241034762ae8aaae296.svg
xlw
user
liwu1995
654db51d0011607c312089c9
Yuzhe Yang
admin_assigned
2023-11-10T11:14:33.679Z
631c3672c73939ffc0714b1f
/avatars/c6f8f69294881532034404ab8e2fe238.svg
yangyuzhe
user
nekoosuki
654db51d0011607c312089ca
Xiang Li
654db51d0011607c312089cb
Yanchun Xie
admin_assigned
2023-11-10T11:15:08.170Z
62a69c16db0b844f8b15e6e1
/avatars/971811e630bd5354a008f6abe3bffcce.svg
yanchun xie
user
xyc
654db51d0011607c312089cc
Yi-Jie Huang
admin_assigned
2023-11-10T11:15:21.281Z
648136af7d65d9ac17346bac
/avatars/7fe65fe712b69f16d81123f0d67719ab.svg
Yijie Huang
user
asdfagafgs
654db51d0011607c312089cd
Yaqian Li
u-LLaVA: Unifying Multi-Modal Tasks via Large Language Model
Recent advances such as LLaVA and Mini-GPT4 have successfully integrated
visual information into LLMs, yielding inspiring outcomes and giving rise to a
new generation of multi-modal LLMs, or MLLMs. Nevertheless, these methods
struggle with hallucinations and the mutual interference between tasks. To
tackle these problems, we propose an efficient and accurate approach to adapt
to downstream tasks by utilizing LLM as a bridge to connect multiple expert
models, namely u-LLaVA. Firstly, we incorporate the modality alignment module
and multi-task modules into LLM. Then, we reorganize or rebuild multi-type
public datasets to enable efficient modality alignment and instruction
following. Finally, task-specific information is extracted from the trained LLM
and provided to different modules for solving downstream tasks. The overall
framework is simple, effective, and achieves state-of-the-art performance
across multiple benchmarks. We also release our model, the generated data, and
the code base publicly available.
A framework u-LLaVA integrates visual information into LLMs using modality alignment and multi-task modules to address hallucinations and task interference, achieving state-of-the-art performance.
LLaVA Mini-GPT4 multi-modal LLMs MLLMs modality alignment module multi-task modules task-specific information