PPLLaVA: Varied Video Sequence Understanding With Prompt Guidance
The past year has witnessed the significant advancement of video-based large
language models. However, the challenge of developing a unified model for both
short and long video understanding remains unresolved. Most existing video LLMs
cannot handle hour-long videos, while methods custom for long videos tend to be
ineffective for shorter videos and images. In this paper, we identify the key
issue as the redundant content in videos. To address this, we propose a novel
pooling strategy that simultaneously achieves token compression and
instruction-aware visual feature aggregation. Our model is termed Prompt-guided
Pooling LLaVA, or PPLLaVA for short. Specifically, PPLLaVA consists of three
core components: the CLIP-based visual-prompt alignment that extracts visual
information relevant to the user's instructions, the prompt-guided pooling that
compresses the visual sequence to arbitrary scales using convolution-style
pooling, and the clip context extension designed for lengthy prompt common in
visual dialogue. Moreover, our codebase also integrates the most advanced video
Direct Preference Optimization (DPO) and visual interleave training. Extensive
experiments have validated the performance of our model. With superior
throughput and only 1024 visual context, PPLLaVA achieves better results on
image benchmarks as a video LLM, while achieving state-of-the-art performance
across various video benchmarks, excelling in tasks ranging from caption
generation to multiple-choice questions, and handling video lengths from
seconds to hours. Codes have been available at
https://github.com/farewellthree/PPLLaVA.
672a17a8341f15009010af9d
Ruyang Liu
claimed_verified
2024-11-05T13:36:20.999Z
650f0e3988cdfe73a864b9c5
/avatars/ce74b96743a99819a762c23b4f2204fd.svg
Ruyang Liu
user
farewellthree
672a17a8341f15009010af9e
Haoran Tang
672a17a8341f15009010af9f
Haibo Liu
672a17a8341f15009010afa0
Yixiao Ge
672a17a8341f15009010afa1
Ying Shan
672a17a8341f15009010afa2
Chen Li
672a17a8341f15009010afa3
Jiankun Yang
PPLLaVA: Varied Video Sequence Understanding With Prompt Guidance
The past year has witnessed the significant advancement of video-based large
language models. However, the challenge of developing a unified model for both
short and long video understanding remains unresolved. Most existing video LLMs
cannot handle hour-long videos, while methods custom for long videos tend to be
ineffective for shorter videos and images. In this paper, we identify the key
issue as the redundant content in videos. To address this, we propose a novel
pooling strategy that simultaneously achieves token compression and
instruction-aware visual feature aggregation. Our model is termed Prompt-guided
Pooling LLaVA, or PPLLaVA for short. Specifically, PPLLaVA consists of three
core components: the CLIP-based visual-prompt alignment that extracts visual
information relevant to the user's instructions, the prompt-guided pooling that
compresses the visual sequence to arbitrary scales using convolution-style
pooling, and the clip context extension designed for lengthy prompt common in
visual dialogue. Moreover, our codebase also integrates the most advanced video
Direct Preference Optimization (DPO) and visual interleave training. Extensive
experiments have validated the performance of our model. With superior
throughput and only 1024 visual context, PPLLaVA achieves better results on
image benchmarks as a video LLM, while achieving state-of-the-art performance
across various video benchmarks, excelling in tasks ranging from caption
generation to multiple-choice questions, and handling video lengths from
seconds to hours. Codes have been available at
https://github.com/farewellthree/PPLLaVA.
PPLLaVA is a video-based large language model that addresses challenges in handling both short and long videos by using a prompt-guided pooling strategy to efficiently compress and aggregate visual features.
video-based large language models token compression instruction-aware visual feature aggregation CLIP-based visual-prompt alignment prompt-guided pooling convolution-style pooling clip context extension Direct Preference Optimization (DPO) visual interleave training caption generation multiple-choice questions