Taking Notes Brings Focus? Towards Multi-Turn Multimodal Dialogue
  Learning
Multimodal large language models (MLLMs), built on large-scale pre-trained
vision towers and language models, have shown great capabilities in multimodal
understanding. However, most existing MLLMs are trained on single-turn vision
question-answering tasks, which do not accurately reflect real-world human
conversations. In this paper, we introduce MMDiag, a multi-turn multimodal
dialogue dataset. This dataset is collaboratively generated through
deliberately designed rules and GPT assistance, featuring strong correlations
between questions, between questions and images, and among different image
regions; thus aligning more closely with real-world scenarios. MMDiag serves as
a strong benchmark for multi-turn multimodal dialogue learning and brings more
challenges to the grounding and reasoning capabilities of MLLMs. Further,
inspired by human vision processing, we present DiagNote, an MLLM equipped with
multimodal grounding and reasoning capabilities. DiagNote consists of two
modules (Deliberate and Gaze) interacting with each other to perform
Chain-of-Thought and annotations respectively, throughout multi-turn dialogues.
We empirically demonstrate the advantages of DiagNote in both grounding and
jointly processing and reasoning with vision and language information over
existing MLLMs.
67cfa814d212c9c5048845a0
Jiazheng Liu
67cfa814d212c9c5048845a1
Sipeng Zheng
admin_assigned
2025-03-11T10:59:45.667Z
64eac1f496f42afd627d439c
/avatars/aa46265122b8a1170f57475494d7922e.svg
Sipeng Zheng
user
sipeng9527
67cfa814d212c9c5048845a2
Börje F. Karlsson
claimed_verified
2025-03-11T08:22:32.095Z
61e52be53d6dbb1da842316a
https://cdn-avatars.huggingface.co/v1/production/uploads/61e52be53d6dbb1da842316a/gx0WGPcOCClXPymoKglc4.jpeg
Börje Karlsson
user
tellarin
67cfa814d212c9c5048845a3
Zongqing Lu
Taking Notes Brings Focus? Towards Multi-Turn Multimodal Dialogue
  Learning
Multimodal large language models (MLLMs), built on large-scale pre-trained
vision towers and language models, have shown great capabilities in multimodal
understanding. However, most existing MLLMs are trained on single-turn vision
question-answering tasks, which do not accurately reflect real-world human
conversations. In this paper, we introduce MMDiag, a multi-turn multimodal
dialogue dataset. This dataset is collaboratively generated through
deliberately designed rules and GPT assistance, featuring strong correlations
between questions, between questions and images, and among different image
regions; thus aligning more closely with real-world scenarios. MMDiag serves as
a strong benchmark for multi-turn multimodal dialogue learning and brings more
challenges to the grounding and reasoning capabilities of MLLMs. Further,
inspired by human vision processing, we present DiagNote, an MLLM equipped with
multimodal grounding and reasoning capabilities. DiagNote consists of two
modules (Deliberate and Gaze) interacting with each other to perform
Chain-of-Thought and annotations respectively, throughout multi-turn dialogues.
We empirically demonstrate the advantages of DiagNote in both grounding and
jointly processing and reasoning with vision and language information over
existing MLLMs.
MMDiag, a multi-turn multimodal dialogue dataset, challenges MLLMs with real-world conversational scenarios, and DiagNote, an MLLM with multimodal grounding and reasoning, outperforms existing models in these tasks.
Multimodal large language models MLLMs vision towers language models multimodal understanding MMDiag multi-turn multimodal dialogue GPT grounding reasoning Chain-of-Thought DiagNote