Video-Panda: Parameter-efficient Alignment for Encoder-free
  Video-Language Models
We present an efficient encoder-free approach for video-language
understanding that achieves competitive performance while significantly
reducing computational overhead. Current video-language models typically rely
on heavyweight image encoders (300M-1.1B parameters) or video encoders (1B-1.4B
parameters), creating a substantial computational burden when processing
multi-frame videos. Our method introduces a novel Spatio-Temporal Alignment
Block (STAB) that directly processes video inputs without requiring pre-trained
encoders while using only 45M parameters for visual processing - at least a
6.5times reduction compared to traditional approaches. The STAB architecture
combines Local Spatio-Temporal Encoding for fine-grained feature extraction,
efficient spatial downsampling through learned attention and separate
mechanisms for modeling frame-level and video-level relationships. Our model
achieves comparable or superior performance to encoder-based approaches for
open-ended video question answering on standard benchmarks. The fine-grained
video question-answering evaluation demonstrates our model's effectiveness,
outperforming the encoder-based approaches Video-ChatGPT and Video-LLaVA in key
aspects like correctness and temporal understanding. Extensive ablation studies
validate our architectural choices and demonstrate the effectiveness of our
spatio-temporal modeling approach while achieving 3-4times faster processing
speeds than previous methods. Code is available at
https://github.com/jh-yi/Video-Panda.
676d854b8771d55751ee0f4f
Jinhui Yi
claimed_verified
2024-12-30T19:31:40.301Z
667dfe6f684550b7893497c5
/avatars/3f48b041a826fd8a8cc74c2d4ced1705.svg
Jinhui Yi
user
jh-yi
676d854b8771d55751ee0f50
Syed Talal Wasim
676d854b8771d55751ee0f51
Yanan Luo
676d854b8771d55751ee0f52
Muzammal Naseer
admin_assigned
2024-12-30T21:17:02.313Z
6495aae2e9b6d89a5c65cf15
/avatars/c176d68af77698745c23da8b92424738.svg
Muzammal Naseer
user
muzammal
676d854b8771d55751ee0f53
Juergen Gall
Video-Panda: Parameter-efficient Alignment for Encoder-free
  Video-Language Models
We present an efficient encoder-free approach for video-language
understanding that achieves competitive performance while significantly
reducing computational overhead. Current video-language models typically rely
on heavyweight image encoders (300M-1.1B parameters) or video encoders (1B-1.4B
parameters), creating a substantial computational burden when processing
multi-frame videos. Our method introduces a novel Spatio-Temporal Alignment
Block (STAB) that directly processes video inputs without requiring pre-trained
encoders while using only 45M parameters for visual processing - at least a
6.5times reduction compared to traditional approaches. The STAB architecture
combines Local Spatio-Temporal Encoding for fine-grained feature extraction,
efficient spatial downsampling through learned attention and separate
mechanisms for modeling frame-level and video-level relationships. Our model
achieves comparable or superior performance to encoder-based approaches for
open-ended video question answering on standard benchmarks. The fine-grained
video question-answering evaluation demonstrates our model's effectiveness,
outperforming the encoder-based approaches Video-ChatGPT and Video-LLaVA in key
aspects like correctness and temporal understanding. Extensive ablation studies
validate our architectural choices and demonstrate the effectiveness of our
spatio-temporal modeling approach while achieving 3-4times faster processing
speeds than previous methods. Code is available at
https://github.com/jh-yi/Video-Panda.
An encoder-free approach for video-language understanding using Spatio-Temporal Alignment Blocks achieves competitive performance with significantly reduced computational overhead and faster processing speeds.
Spatio-Temporal Alignment Block STAB Local Spatio-Temporal Encoding learned attention frame-level relationships video-level relationships video question answering Video-ChatGPT Video-LLaVA