LLaDA-V: Large Language Diffusion Models with Visual Instruction Tuning
In this work, we introduce LLaDA-V, a purely diffusion-based Multimodal Large
Language Model (MLLM) that integrates visual instruction tuning with masked
diffusion models, representing a departure from the autoregressive paradigms
dominant in current multimodal approaches. Built upon LLaDA, a representative
large language diffusion model, LLaDA-V incorporates a vision encoder and MLP
connector that projects visual features into the language embedding space,
enabling effective multimodal alignment. Our empirical investigation reveals
several intriguing results: First, LLaDA-V demonstrates promising multimodal
performance despite its language model being weaker on purely textual tasks
than counterparts like LLaMA3-8B and Qwen2-7B. When trained on the same
instruction data, LLaDA-V is highly competitive to LLaMA3-V across multimodal
tasks with better data scalability. It also narrows the performance gap to
Qwen2-VL, suggesting the effectiveness of its architecture for multimodal
tasks. Second, LLaDA-V achieves state-of-the-art performance in multimodal
understanding compared to existing hybrid autoregressive-diffusion and purely
diffusion-based MLLMs. Our findings suggest that large language diffusion
models show promise in multimodal contexts and warrant further investigation in
future research. Project page and codes:
https://ml-gsai.github.io/LLaDA-V-demo/.
682fe37bb998c9f79463b563
Zebin You
admin_assigned
2025-05-27T08:36:23.049Z
624f909eac5dd186b01ac3f5
/avatars/0aafdb1cbb492fda52a0303031cc6c14.svg
Zebin You
user
yyyou
682fe37bb998c9f79463b564
Shen Nie
682fe37bb998c9f79463b565
Xiaolu Zhang
admin_assigned
2025-05-27T08:36:49.675Z
67513d6d3b8586521cda5d76
/avatars/0f95cc5c23a0a1da289aa785bd33b616.svg
Xiaolu  Zhang
user
xiaolu0714
682fe37bb998c9f79463b566
Jun Hu
682fe37bb998c9f79463b567
Jun Zhou
682fe37bb998c9f79463b568
Zhiwu Lu
admin_assigned
2025-05-27T08:37:09.401Z
6351e3035b904878f1d36719
/avatars/a7b3a5faacaa483fab1a5b1b95bcc0a4.svg
Zhiwu Lu
user
zewlu
682fe37bb998c9f79463b569
Ji-Rong Wen
admin_assigned
2025-05-27T08:37:03.209Z
64b8c89052b7353d8c6a1013
/avatars/cd59fffe81f6b07b4519540b8ff3d95f.svg
Ji-Rong Wen
user
jrwen
682fe37bb998c9f79463b56a
Chongxuan Li
admin_assigned
2025-05-27T08:36:56.958Z
64c07b488e2612254361153b
/avatars/ade0f783cc4c2d3e73f402637f595471.svg
chongxuan li
user
zhenxuan00
LLaDA-V: Large Language Diffusion Models with Visual Instruction Tuning
In this work, we introduce LLaDA-V, a purely diffusion-based Multimodal Large
Language Model (MLLM) that integrates visual instruction tuning with masked
diffusion models, representing a departure from the autoregressive paradigms
dominant in current multimodal approaches. Built upon LLaDA, a representative
large language diffusion model, LLaDA-V incorporates a vision encoder and MLP
connector that projects visual features into the language embedding space,
enabling effective multimodal alignment. Our empirical investigation reveals
several intriguing results: First, LLaDA-V demonstrates promising multimodal
performance despite its language model being weaker on purely textual tasks
than counterparts like LLaMA3-8B and Qwen2-7B. When trained on the same
instruction data, LLaDA-V is highly competitive to LLaMA3-V across multimodal
tasks with better data scalability. It also narrows the performance gap to
Qwen2-VL, suggesting the effectiveness of its architecture for multimodal
tasks. Second, LLaDA-V achieves state-of-the-art performance in multimodal
understanding compared to existing hybrid autoregressive-diffusion and purely
diffusion-based MLLMs. Our findings suggest that large language diffusion
models show promise in multimodal contexts and warrant further investigation in
future research. Project page and codes:
https://ml-gsai.github.io/LLaDA-V-demo/.
A diffusion-based Multimodal Large Language Model (LLaDA-V) with integrated visual instruction tuning performs competitively on multimodal tasks and outperforms existing models in multimodal understanding.
diffusion-based Multimodal Large Language Model (MLLM) visual instruction tuning masked diffusion models autoregressive paradigms vision encoder MLP connector language embedding space multimodal performance LLaDA LLaMA3-8B Qwen2-7B LLaMA3-V Qwen2-VL multimodal understanding hybrid autoregressive-diffusion