PLLaVA : Parameter-free LLaVA Extension from Images to Videos for Video
  Dense Captioning
Vision-language pre-training has significantly elevated performance across a
wide range of image-language applications. Yet, the pre-training process for
video-related tasks demands exceptionally large computational and data
resources, which hinders the progress of video-language models. This paper
investigates a straightforward, highly efficient, and resource-light approach
to adapting an existing image-language pre-trained model for dense video
understanding. Our preliminary experiments reveal that directly fine-tuning
pre-trained image-language models with multiple frames as inputs on video
datasets leads to performance saturation or even a drop. Our further
investigation reveals that it is largely attributed to the bias of learned
high-norm visual features. Motivated by this finding, we propose a simple but
effective pooling strategy to smooth the feature distribution along the
temporal dimension and thus reduce the dominant impacts from the extreme
features. The new model is termed Pooling LLaVA, or  in short.
 achieves new state-of-the-art performance on modern benchmark
datasets for both video question-answer and captioning tasks. Notably, on the
recent popular Video ChatGPT benchmark, PLLaVA achieves a score of 3.48 out of
5 on average of five evaluated dimensions, exceeding the previous SOTA results
from GPT4V (IG-VLM) by 9\%. On the latest multi-choice benchmark MVBench,
PLLaVA achieves 58.1\% accuracy on average across 20 sub-tasks, 14.5\% higher
than GPT4V (IG-VLM). Code is available at
https://github.com/magic-research/PLLaVA.
662ef3e7b10f82aa4fabd7c9
Lin Xu
662ef3e7b10f82aa4fabd7ca
Yilin Zhao
admin_assigned
2024-04-29T11:48:54.834Z
6385d86181fe8c678a345ff0
https://cdn-avatars.huggingface.co/v1/production/uploads/6385d86181fe8c678a345ff0/ETvVOtBAQ-RpwfT4TCWsv.jpeg
Yilin Zhao
user
ermu2001
662ef3e7b10f82aa4fabd7cb
Daquan Zhou
admin_assigned
2024-04-29T11:49:16.844Z
63fe0b160c1bbe8e29d2dd32
/avatars/bc574036287170a77057893efaa48e2d.svg
Zhou
user
DaQuan21
662ef3e7b10f82aa4fabd7cc
Zhijie Lin
admin_assigned
2024-04-29T11:57:19.161Z
64415957bd0c9726529802f6
/avatars/1132d1ee68fb58ec635d57c8175caacd.svg
Zhijie Lin
user
Ikuinen
662ef3e7b10f82aa4fabd7cd
See Kiong Ng
662ef3e7b10f82aa4fabd7ce
Jiashi Feng
PLLaVA : Parameter-free LLaVA Extension from Images to Videos for Video
  Dense Captioning
Vision-language pre-training has significantly elevated performance across a
wide range of image-language applications. Yet, the pre-training process for
video-related tasks demands exceptionally large computational and data
resources, which hinders the progress of video-language models. This paper
investigates a straightforward, highly efficient, and resource-light approach
to adapting an existing image-language pre-trained model for dense video
understanding. Our preliminary experiments reveal that directly fine-tuning
pre-trained image-language models with multiple frames as inputs on video
datasets leads to performance saturation or even a drop. Our further
investigation reveals that it is largely attributed to the bias of learned
high-norm visual features. Motivated by this finding, we propose a simple but
effective pooling strategy to smooth the feature distribution along the
temporal dimension and thus reduce the dominant impacts from the extreme
features. The new model is termed Pooling LLaVA, or  in short.
 achieves new state-of-the-art performance on modern benchmark
datasets for both video question-answer and captioning tasks. Notably, on the
recent popular Video ChatGPT benchmark, PLLaVA achieves a score of 3.48 out of
5 on average of five evaluated dimensions, exceeding the previous SOTA results
from GPT4V (IG-VLM) by 9\%. On the latest multi-choice benchmark MVBench,
PLLaVA achieves 58.1\% accuracy on average across 20 sub-tasks, 14.5\% higher
than GPT4V (IG-VLM). Code is available at
https://github.com/magic-research/PLLaVA.
Pooling LLaVA improves video-language pre-trained models by smoothing high-norm visual features, achieving state-of-the-art performance in video question-answering, captioning, and benchmark tasks.
vision-language pre-training dense video understanding fine-tuning image-language models video datasets high-norm visual features pooling strategy temporal dimension extreme features video question-answer captioning Video ChatGPT benchmark MVBench benchmark accuracy