SlowFast-LLaVA: A Strong Training-Free Baseline for Video Large Language
  Models
We propose SlowFast-LLaVA (or SF-LLaVA for short), a training-free video
large language model (LLM) that can jointly capture the detailed spatial
semantics and long-range temporal context without exceeding the token budget of
commonly used LLMs. This is realized by using a two-stream SlowFast design of
inputs for Video LLMs to aggregate features from sampled video frames in an
effective way. Specifically, the Slow pathway extracts features at a low frame
rate while keeping as many spatial details as possible (e.g., with 24x24
tokens), and the Fast pathway operates on a high frame rate but uses a larger
spatial pooling stride (e.g., downsampling 6x) to focus on the motion cues. As
a result, this design allows us to adequately capture both spatial and temporal
features that are beneficial for understanding details along the video.
Experimental results show that SF-LLaVA outperforms existing training-free
methods on a wide range of video tasks. On some benchmarks, it achieves
comparable or even better performance compared to state-of-the-art Video LLMs
that are fine-tuned on video datasets.
669f21b88a6fac322926ded2
Mingze Xu
claimed_verified
2024-07-24T07:27:25.835Z
65fc655930aee594d74dd7b8
/avatars/e8df7170fa1cf99bb3a2b9a083ee99a1.svg
Mingze Xu
user
xumingze0308
669f21b88a6fac322926ded3
Mingfei Gao
claimed_verified
2024-07-24T07:27:23.525Z
64388d5f1efe72ba48071c52
/avatars/2653fba13e71f471e9d55c134fe25efc.svg
Mingfei Gao
user
fly6464
669f21b88a6fac322926ded4
Zhe Gan
admin_assigned
2024-07-23T08:01:34.704Z
644022db21ec7c165d848b6e
/avatars/d9ed37ec77f2682878b2976a04d2fcd8.svg
Zhe Gan
user
zhegan27
669f21b88a6fac322926ded5
Hong-You Chen
admin_assigned
2024-07-23T08:01:41.355Z
64b720891a4d97b5d7f055b8
/avatars/7247a4f3a433a58fe934f245d9a757b2.svg
Hong-You Chen
user
Cbbjames
669f21b88a6fac322926ded6
Zhengfeng Lai
claimed_verified
2024-08-09T07:48:33.573Z
66b5295f83425904fa7a1a6a
/avatars/a35568fb933ceef7451bd88fb3d5ab17.svg
Zhengfeng Lai
user
jefflai
669f21b88a6fac322926ded7
Haiming Gang
669f21b88a6fac322926ded8
Kai Kang
669f21b88a6fac322926ded9
Afshin Dehghan
SlowFast-LLaVA: A Strong Training-Free Baseline for Video Large Language
  Models
We propose SlowFast-LLaVA (or SF-LLaVA for short), a training-free video
large language model (LLM) that can jointly capture the detailed spatial
semantics and long-range temporal context without exceeding the token budget of
commonly used LLMs. This is realized by using a two-stream SlowFast design of
inputs for Video LLMs to aggregate features from sampled video frames in an
effective way. Specifically, the Slow pathway extracts features at a low frame
rate while keeping as many spatial details as possible (e.g., with 24x24
tokens), and the Fast pathway operates on a high frame rate but uses a larger
spatial pooling stride (e.g., downsampling 6x) to focus on the motion cues. As
a result, this design allows us to adequately capture both spatial and temporal
features that are beneficial for understanding details along the video.
Experimental results show that SF-LLaVA outperforms existing training-free
methods on a wide range of video tasks. On some benchmarks, it achieves
comparable or even better performance compared to state-of-the-art Video LLMs
that are fine-tuned on video datasets.
SlowFast-LLaVA captures detailed spatial and temporal features in videos using a two-stream design, outperforming existing training-free methods and matching fine-tuned Video LLMs.
SlowFast video large language model two-stream design Slow pathway Fast pathway frame rate spatial details temporal context motion cues training-free video tasks state-of-the-art Video LLMs