VideoGLaMM: A Large Multimodal Model for Pixel-Level Visual Grounding in
  Videos
Fine-grained alignment between videos and text is challenging due to complex
spatial and temporal dynamics in videos. Existing video-based Large Multimodal
Models (LMMs) handle basic conversations but struggle with precise pixel-level
grounding in videos. To address this, we introduce VideoGLaMM, a LMM designed
for fine-grained pixel-level grounding in videos based on user-provided textual
inputs. Our design seamlessly connects three key components: a Large Language
Model, a dual vision encoder that emphasizes both spatial and temporal details,
and a spatio-temporal decoder for accurate mask generation. This connection is
facilitated via tunable V-L and L-V adapters that enable close Vision-Language
(VL) alignment. The architecture is trained to synchronize both spatial and
temporal elements of video content with textual instructions. To enable
fine-grained grounding, we curate a multimodal dataset featuring detailed
visually-grounded conversations using a semiautomatic annotation pipeline,
resulting in a diverse set of 38k video-QA triplets along with 83k objects and
671k masks. We evaluate VideoGLaMM on three challenging tasks: Grounded
Conversation Generation, Visual Grounding, and Referring Video Segmentation.
Experimental results show that our model consistently outperforms existing
approaches across all three tasks.
672de757915fdfa0d046b1b3
Shehan Munasinghe
672de757915fdfa0d046b1b4
Hanan Gani
claimed_verified
2024-11-15T09:29:11.665Z
64eddf17e630ae6d575f6231
https://cdn-avatars.huggingface.co/v1/production/uploads/64eddf17e630ae6d575f6231/W8oofVqStLEAK1Xi4fQTh.png
Hanan Gani
user
hanangani
672de757915fdfa0d046b1b5
Wenqi Zhu
672de757915fdfa0d046b1b6
Jiale Cao
672de757915fdfa0d046b1b7
Eric Xing
672de757915fdfa0d046b1b8
Fahad Shahbaz Khan
672de757915fdfa0d046b1b9
Salman Khan
VideoGLaMM: A Large Multimodal Model for Pixel-Level Visual Grounding in
  Videos
Fine-grained alignment between videos and text is challenging due to complex
spatial and temporal dynamics in videos. Existing video-based Large Multimodal
Models (LMMs) handle basic conversations but struggle with precise pixel-level
grounding in videos. To address this, we introduce VideoGLaMM, a LMM designed
for fine-grained pixel-level grounding in videos based on user-provided textual
inputs. Our design seamlessly connects three key components: a Large Language
Model, a dual vision encoder that emphasizes both spatial and temporal details,
and a spatio-temporal decoder for accurate mask generation. This connection is
facilitated via tunable V-L and L-V adapters that enable close Vision-Language
(VL) alignment. The architecture is trained to synchronize both spatial and
temporal elements of video content with textual instructions. To enable
fine-grained grounding, we curate a multimodal dataset featuring detailed
visually-grounded conversations using a semiautomatic annotation pipeline,
resulting in a diverse set of 38k video-QA triplets along with 83k objects and
671k masks. We evaluate VideoGLaMM on three challenging tasks: Grounded
Conversation Generation, Visual Grounding, and Referring Video Segmentation.
Experimental results show that our model consistently outperforms existing
approaches across all three tasks.
VideoGLaMM, a Large Multimodal Model, achieves precise pixel-level grounding in videos using a dual vision encoder, spatio-temporal decoder, and tunable Visual-Language adapters, outperforming existing approaches on various multimodal tasks.
Large Multimodal Models Large Language Model dual vision encoder spatio-temporal decoder V-L and L-V adapters Vision-Language alignment semiautomatic annotation pipeline Grounded Conversation Generation Visual Grounding Referring Video Segmentation