DreamLLM: Synergistic Multimodal Comprehension and Creation
This paper presents DreamLLM, a learning framework that first achieves
versatile Multimodal Large Language Models (MLLMs) empowered with frequently
overlooked synergy between multimodal comprehension and creation. DreamLLM
operates on two fundamental principles. The first focuses on the generative
modeling of both language and image posteriors by direct sampling in the raw
multimodal space. This approach circumvents the limitations and information
loss inherent to external feature extractors like CLIP, and a more thorough
multimodal understanding is obtained. Second, DreamLLM fosters the generation
of raw, interleaved documents, modeling both text and image contents, along
with unstructured layouts. This allows DreamLLM to learn all conditional,
marginal, and joint multimodal distributions effectively. As a result, DreamLLM
is the first MLLM capable of generating free-form interleaved content.
Comprehensive experiments highlight DreamLLM's superior performance as a
zero-shot multimodal generalist, reaping from the enhanced learning synergy.
650bc2b324da30e773ea9762
Runpei Dong
admin_assigned
2023-09-21T07:46:39.028Z
6201fc5d91d53938a6432fbf
https://cdn-avatars.huggingface.co/v1/production/uploads/6201fc5d91d53938a6432fbf/VLs8ZYaZrop4KBpZn53fH.jpeg
Runpei Dong
user
RunpeiDong
650bc2b324da30e773ea9763
Chunrui Han
650bc2b324da30e773ea9764
Yuang Peng
claimed_verified
2023-09-21T07:22:38.366Z
631ee086c1a8269da39265c6
https://cdn-avatars.huggingface.co/v1/production/uploads/631ee086c1a8269da39265c6/wUa1epGtTGcUv2mvrLUcD.png
Yuang Peng
user
yuangpeng
650bc2b324da30e773ea9765
Zekun Qi
admin_assigned
2023-09-21T07:47:48.699Z
63c3e8abc7d7f4c63a515a02
https://cdn-avatars.huggingface.co/v1/production/uploads/63c3e8abc7d7f4c63a515a02/npMHnVP2hHLbvoUGe7C4O.jpeg
Zekun Qi
user
qizekun
650bc2b324da30e773ea9766
Zheng Ge
650bc2b324da30e773ea9767
Jinrong Yang
admin_assigned
2023-09-21T09:26:48.880Z
650bd10c1f2949b99e65ef23
/avatars/8f36f844690b64bcc7c358063f2bdc3b.svg
Jinrong Yang
user
YancieYJR
650bc2b324da30e773ea9768
Liang Zhao
650bc2b324da30e773ea9769
Jianjian Sun
650bc2b324da30e773ea976a
Hongyu Zhou
650bc2b324da30e773ea976b
Haoran Wei
claimed_verified
2024-01-24T08:09:54.188Z
6436618aeef1f55654a9f458
https://cdn-avatars.huggingface.co/v1/production/uploads/6436618aeef1f55654a9f458/OvxGtuDg2GAFG9As-2hzW.jpeg
Haoran Wei
user
HaoranWei
650bc2b324da30e773ea976c
Xiangwen Kong
650bc2b324da30e773ea976d
Xiangyu Zhang
650bc2b324da30e773ea976e
Kaisheng Ma
650bc2b324da30e773ea976f
Li Yi
DreamLLM: Synergistic Multimodal Comprehension and Creation
This paper presents DreamLLM, a learning framework that first achieves
versatile Multimodal Large Language Models (MLLMs) empowered with frequently
overlooked synergy between multimodal comprehension and creation. DreamLLM
operates on two fundamental principles. The first focuses on the generative
modeling of both language and image posteriors by direct sampling in the raw
multimodal space. This approach circumvents the limitations and information
loss inherent to external feature extractors like CLIP, and a more thorough
multimodal understanding is obtained. Second, DreamLLM fosters the generation
of raw, interleaved documents, modeling both text and image contents, along
with unstructured layouts. This allows DreamLLM to learn all conditional,
marginal, and joint multimodal distributions effectively. As a result, DreamLLM
is the first MLLM capable of generating free-form interleaved content.
Comprehensive experiments highlight DreamLLM's superior performance as a
zero-shot multimodal generalist, reaping from the enhanced learning synergy.
DreamLLM, a framework for Multimodal Large Language Models, directly samples in the multimodal space to enhance comprehension and creation synergy, enabling free-form interleaved content generation.
generative modeling multimodal space CLIP feature extractors interleaved documents multimodal distributions zero-shot multimodal generalist