Summarization of Multimodal Presentations with Vision-Language Models:
  Study of the Effect of Modalities and Structure
Vision-Language Models (VLMs) can process visual and textual information in
multiple formats: texts, images, interleaved texts and images, or even
hour-long videos. In this work, we conduct fine-grained quantitative and
qualitative analyses of automatic summarization of multimodal presentations
using VLMs with various representations as input. From these experiments, we
suggest cost-effective strategies for generating summaries from text-heavy
multimodal documents under different input-length budgets using VLMs. We show
that slides extracted from the video stream can be beneficially used as input
against the raw video, and that a structured representation from interleaved
slides and transcript provides the best performance. Finally, we reflect and
comment on the nature of cross-modal interactions in multimodal presentations
and share suggestions to improve the capabilities of VLMs to understand
documents of this nature.
67ff5b335cf0fe153845d1c9
Théo Gigant
claimed_verified
2025-04-16T08:42:49.462Z
60d35154d7b174177faabd55
https://cdn-avatars.huggingface.co/v1/production/uploads/1645712223620-60d35154d7b174177faabd55.jpeg
Théo Gigant
user
gigant
67ff5b335cf0fe153845d1ca
Camille Guinaudeau
67ff5b335cf0fe153845d1cb
Frédéric Dufaux
Summarization of Multimodal Presentations with Vision-Language Models:
  Study of the Effect of Modalities and Structure
Vision-Language Models (VLMs) can process visual and textual information in
multiple formats: texts, images, interleaved texts and images, or even
hour-long videos. In this work, we conduct fine-grained quantitative and
qualitative analyses of automatic summarization of multimodal presentations
using VLMs with various representations as input. From these experiments, we
suggest cost-effective strategies for generating summaries from text-heavy
multimodal documents under different input-length budgets using VLMs. We show
that slides extracted from the video stream can be beneficially used as input
against the raw video, and that a structured representation from interleaved
slides and transcript provides the best performance. Finally, we reflect and
comment on the nature of cross-modal interactions in multimodal presentations
and share suggestions to improve the capabilities of VLMs to understand
documents of this nature.
Analyses of VLMs for automatic summarization of multimodal presentations demonstrate that structured representations of interleaved slides and transcripts yield the best performance.
Vision-Language Models VLMs automatic summarization multimodal presentations slides transcript cross-modal interactions