PG-Video-LLaVA: Pixel Grounding Large Video-Language Models
Extending image-based Large Multimodal Models (LMM) to videos is challenging
due to the inherent complexity of video data. The recent approaches extending
image-based LMM to videos either lack the grounding capabilities (e.g.,
VideoChat, Video-ChatGPT, Video-LLaMA) or do not utilize the audio-signals for
better video understanding (e.g., Video-ChatGPT). Addressing these gaps, we
propose Video-LLaVA, the first LMM with pixel-level grounding capability,
integrating audio cues by transcribing them into text to enrich video-context
understanding. Our framework uses an off-the-shelf tracker and a novel
grounding module, enabling it to spatially and temporally localize objects in
videos following user instructions. We evaluate Video-LLaVA using video-based
generative and question-answering benchmarks and introduce new benchmarks
specifically designed to measure prompt-based object grounding performance in
videos. Further, we propose the use of Vicuna over GPT-3.5, as utilized in
Video-ChatGPT, for video-based conversation benchmarking, ensuring
reproducibility of results which is a concern with the proprietary nature of
GPT-3.5. Our framework builds on SoTA image-based LLaVA model and extends its
advantages to the video domain, delivering promising gains on video-based
conversation and grounding tasks. Project Page:
https://github.com/mbzuai-oryx/Video-LLaVA
655edf70b9673a4249b65f7f
Shehan Munasinghe
admin_assigned
2023-11-23T16:14:44.171Z
641fecb25bccaa42483db102
https://cdn-avatars.huggingface.co/v1/production/uploads/641fecb25bccaa42483db102/HPTuSuWyIBHgg9afcKZ7p.jpeg
Shehan Munasinghe
user
shehan97
655edf70b9673a4249b65f80
Rusiru Thushara
admin_assigned
2023-11-23T16:15:04.182Z
64fda5aeda4429f25d2b0479
/avatars/4d8d32189b0bd5b585a5ea7c79226e3f.svg
Rusiru Thushara
user
thusharakart
655edf70b9673a4249b65f81
Muhammad Maaz
claimed_verified
2023-11-24T08:45:11.563Z
64807585856901b0edb8d68b
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/kmCo6brXhQ7SyxVx3lpsx.jpeg
Muhammad Maaz
user
mmaaz60
655edf70b9673a4249b65f82
Hanoona Abdul Rasheed
admin_assigned
2023-11-23T16:15:36.566Z
64636b2551fa6e6306046293
https://cdn-avatars.huggingface.co/v1/production/uploads/64636b2551fa6e6306046293/Uuz6z2MZb_LKLGM8uxF9s.jpeg
Hanoona Rasheed
user
Hanoona
655edf70b9673a4249b65f83
Salman Khan
655edf70b9673a4249b65f84
Mubarak Shah
655edf70b9673a4249b65f85
Fahad Khan
PG-Video-LLaVA: Pixel Grounding Large Video-Language Models
Extending image-based Large Multimodal Models (LMM) to videos is challenging
due to the inherent complexity of video data. The recent approaches extending
image-based LMM to videos either lack the grounding capabilities (e.g.,
VideoChat, Video-ChatGPT, Video-LLaMA) or do not utilize the audio-signals for
better video understanding (e.g., Video-ChatGPT). Addressing these gaps, we
propose Video-LLaVA, the first LMM with pixel-level grounding capability,
integrating audio cues by transcribing them into text to enrich video-context
understanding. Our framework uses an off-the-shelf tracker and a novel
grounding module, enabling it to spatially and temporally localize objects in
videos following user instructions. We evaluate Video-LLaVA using video-based
generative and question-answering benchmarks and introduce new benchmarks
specifically designed to measure prompt-based object grounding performance in
videos. Further, we propose the use of Vicuna over GPT-3.5, as utilized in
Video-ChatGPT, for video-based conversation benchmarking, ensuring
reproducibility of results which is a concern with the proprietary nature of
GPT-3.5. Our framework builds on SoTA image-based LLaVA model and extends its
advantages to the video domain, delivering promising gains on video-based
conversation and grounding tasks. Project Page:
https://github.com/mbzuai-oryx/Video-LLaVA
Video-LLaVA integrates audio transcriptions and uses a novel grounding module to enhance pixel-level grounding in large multimodal video models, leading to improved performance in generative and question-answering tasks.
Large Multimodal Models LMM video-based LMM Video-ChatGPT Video-LLaMA pixel-level grounding audio cues text transcription object localization user instructions video-based generative benchmarks video-based question-answering benchmarks Vicuna GPT-3.5 SoTA LLaVA model