Summarization of Multimodal Presentations with Vision-Language Models:
  Study of the Effect of Modalities and Structure
Vision-Language Models (VLMs) can process visual and textual information in
multiple formats: texts, images, interleaved texts and images, or even
hour-long videos. In this work, we conduct fine-grained quantitative and
qualitative analyses of automatic summarization of multimodal presentations
using VLMs with various representations as input. From these experiments, we
suggest cost-effective strategies for generating summaries from text-heavy
multimodal documents under different input-length budgets using VLMs. We show
that slides extracted from the video stream can be beneficially used as input
against the raw video, and that a structured representation from interleaved
slides and transcript provides the best performance. Finally, we reflect and
comment on the nature of cross-modal interactions in multimodal presentations
and share suggestions to improve the capabilities of VLMs to understand
documents of this nature.
67ff5b335cf0fe153845d1c9
Théo Gigant
claimed_verified
2025-04-16T08:42:49.462Z
60d35154d7b174177faabd55
https://cdn-avatars.huggingface.co/v1/production/uploads/1645712223620-60d35154d7b174177faabd55.jpeg
Théo Gigant
user
gigant
67ff5b335cf0fe153845d1ca
Camille Guinaudeau
67ff5b335cf0fe153845d1cb
Frédéric Dufaux
Summarization of Multimodal Presentations with Vision-Language Models:
  Study of the Effect of Modalities and Structure
Vision-Language Models (VLMs) can process visual and textual information in
multiple formats: texts, images, interleaved texts and images, or even
hour-long videos. In this work, we conduct fine-grained quantitative and
qualitative analyses of automatic summarization of multimodal presentations
using VLMs with various representations as input. From these experiments, we
suggest cost-effective strategies for generating summaries from text-heavy
multimodal documents under different input-length budgets using VLMs. We show
that slides extracted from the video stream can be beneficially used as input
against the raw video, and that a structured representation from interleaved
slides and transcript provides the best performance. Finally, we reflect and
comment on the nature of cross-modal interactions in multimodal presentations
and share suggestions to improve the capabilities of VLMs to understand
documents of this nature.
Vision-Language Models (VLMs) automatic summarization multimodal presentations text-heavy multimodal documents input-length budgets slides video stream raw video structured representation interleaved slides transcript cross-modal interactions