Multimodal DeepResearcher: Generating Text-Chart Interleaved Reports
  From Scratch with Agentic Framework
Visualizations play a crucial part in effective communication of concepts and
information. Recent advances in reasoning and retrieval augmented generation
have enabled Large Language Models (LLMs) to perform deep research and generate
comprehensive reports. Despite its progress, existing deep research frameworks
primarily focus on generating text-only content, leaving the automated
generation of interleaved texts and visualizations underexplored. This novel
task poses key challenges in designing informative visualizations and
effectively integrating them with text reports. To address these challenges, we
propose Formal Description of Visualization (FDV), a structured textual
representation of charts that enables LLMs to learn from and generate diverse,
high-quality visualizations. Building on this representation, we introduce
Multimodal DeepResearcher, an agentic framework that decomposes the task into
four stages: (1) researching, (2) exemplar report textualization, (3) planning,
and (4) multimodal report generation. For the evaluation of generated
multimodal reports, we develop MultimodalReportBench, which contains 100
diverse topics served as inputs along with 5 dedicated metrics. Extensive
experiments across models and evaluation methods demonstrate the effectiveness
of Multimodal DeepResearcher. Notably, utilizing the same Claude 3.7 Sonnet
model, Multimodal DeepResearcher achieves an 82\% overall win rate over the
baseline method.
683fb5d592425f86f2be5c40
Zhaorui Yang
admin_assigned
2025-06-05T07:37:44.850Z
64a568f5764b1dce366f9fd2
https://cdn-avatars.huggingface.co/v1/production/uploads/64a568f5764b1dce366f9fd2/9THW2AJhEVmzltEAm9mMY.jpeg
Zhaorui Yang
user
zhaoruiyang
683fb5d592425f86f2be5c41
Bo Pan
683fb5d592425f86f2be5c42
Han Wang
683fb5d592425f86f2be5c43
Yiyao Wang
683fb5d592425f86f2be5c44
Xingyu Liu
683fb5d592425f86f2be5c45
Minfeng Zhu
admin_assigned
2025-06-05T07:37:53.514Z
66771ddf6a685a4ffb0fbf36
https://cdn-avatars.huggingface.co/v1/production/uploads/66771ddf6a685a4ffb0fbf36/Ucp9z_Av4LwyR1TEz2jz_.jpeg
Minfeng Zhu
user
twilightsnow
683fb5d592425f86f2be5c46
Bo Zhang
683fb5d592425f86f2be5c47
Wei Chen
Multimodal DeepResearcher: Generating Text-Chart Interleaved Reports
  From Scratch with Agentic Framework
Visualizations play a crucial part in effective communication of concepts and
information. Recent advances in reasoning and retrieval augmented generation
have enabled Large Language Models (LLMs) to perform deep research and generate
comprehensive reports. Despite its progress, existing deep research frameworks
primarily focus on generating text-only content, leaving the automated
generation of interleaved texts and visualizations underexplored. This novel
task poses key challenges in designing informative visualizations and
effectively integrating them with text reports. To address these challenges, we
propose Formal Description of Visualization (FDV), a structured textual
representation of charts that enables LLMs to learn from and generate diverse,
high-quality visualizations. Building on this representation, we introduce
Multimodal DeepResearcher, an agentic framework that decomposes the task into
four stages: (1) researching, (2) exemplar report textualization, (3) planning,
and (4) multimodal report generation. For the evaluation of generated
multimodal reports, we develop MultimodalReportBench, which contains 100
diverse topics served as inputs along with 5 dedicated metrics. Extensive
experiments across models and evaluation methods demonstrate the effectiveness
of Multimodal DeepResearcher. Notably, utilizing the same Claude 3.7 Sonnet
model, Multimodal DeepResearcher achieves an 82\% overall win rate over the
baseline method.
A new framework, Multimodal DeepResearcher, enables Large Language Models to generate high-quality multimodal reports combining text and diverse visualizations through structured textual representations.
Formal Description of Visualization FDV Multimodal DeepResearcher researching exemplar report textualization planning multimodal report generation MultimodalReportBench multimodal reports