VideoGLaMM: A Large Multimodal Model for Pixel-Level Visual Grounding in
  Videos
Fine-grained alignment between videos and text is challenging due to complex
spatial and temporal dynamics in videos. Existing video-based Large Multimodal
Models (LMMs) handle basic conversations but struggle with precise pixel-level
grounding in videos. To address this, we introduce VideoGLaMM, a LMM designed
for fine-grained pixel-level grounding in videos based on user-provided textual
inputs. Our design seamlessly connects three key components: a Large Language
Model, a dual vision encoder that emphasizes both spatial and temporal details,
and a spatio-temporal decoder for accurate mask generation. This connection is
facilitated via tunable V-L and L-V adapters that enable close Vision-Language
(VL) alignment. The architecture is trained to synchronize both spatial and
temporal elements of video content with textual instructions. To enable
fine-grained grounding, we curate a multimodal dataset featuring detailed
visually-grounded conversations using a semiautomatic annotation pipeline,
resulting in a diverse set of 38k video-QA triplets along with 83k objects and
671k masks. We evaluate VideoGLaMM on three challenging tasks: Grounded
Conversation Generation, Visual Grounding, and Referring Video Segmentation.
Experimental results show that our model consistently outperforms existing
approaches across all three tasks.
672de757915fdfa0d046b1b3
Shehan Munasinghe
672de757915fdfa0d046b1b4
Hanan Gani
claimed_verified
2024-11-15T09:29:11.665Z
64eddf17e630ae6d575f6231
https://cdn-avatars.huggingface.co/v1/production/uploads/64eddf17e630ae6d575f6231/W8oofVqStLEAK1Xi4fQTh.png
Hanan Gani
user
hanangani
672de757915fdfa0d046b1b5
Wenqi Zhu
672de757915fdfa0d046b1b6
Jiale Cao
672de757915fdfa0d046b1b7
Eric Xing
672de757915fdfa0d046b1b8
Fahad Shahbaz Khan
672de757915fdfa0d046b1b9
Salman Khan
VideoGLaMM: A Large Multimodal Model for Pixel-Level Visual Grounding in
  Videos
Fine-grained alignment between videos and text is challenging due to complex
spatial and temporal dynamics in videos. Existing video-based Large Multimodal
Models (LMMs) handle basic conversations but struggle with precise pixel-level
grounding in videos. To address this, we introduce VideoGLaMM, a LMM designed
for fine-grained pixel-level grounding in videos based on user-provided textual
inputs. Our design seamlessly connects three key components: a Large Language
Model, a dual vision encoder that emphasizes both spatial and temporal details,
and a spatio-temporal decoder for accurate mask generation. This connection is
facilitated via tunable V-L and L-V adapters that enable close Vision-Language
(VL) alignment. The architecture is trained to synchronize both spatial and
temporal elements of video content with textual instructions. To enable
fine-grained grounding, we curate a multimodal dataset featuring detailed
visually-grounded conversations using a semiautomatic annotation pipeline,
resulting in a diverse set of 38k video-QA triplets along with 83k objects and
671k masks. We evaluate VideoGLaMM on three challenging tasks: Grounded
Conversation Generation, Visual Grounding, and Referring Video Segmentation.
Experimental results show that our model consistently outperforms existing
approaches across all three tasks.
VideoGLaMM Large Multimodal Models (LMMs) Large Language Model dual vision encoder spatio-temporal decoder tunable V-L and L-V adapters Vision-Language (VL) alignment fine-grained pixel-level grounding Video-QA triplets Grounded Conversation Generation Visual Grounding Referring Video Segmentation