VISTA: Enhancing Long-Duration and High-Resolution Video Understanding
  by Video Spatiotemporal Augmentation
Current large multimodal models (LMMs) face significant challenges in
processing and comprehending long-duration or high-resolution videos, which is
mainly due to the lack of high-quality datasets. To address this issue from a
data-centric perspective, we propose VISTA, a simple yet effective Video
Spatiotemporal Augmentation framework that synthesizes long-duration and
high-resolution video instruction-following pairs from existing video-caption
datasets. VISTA spatially and temporally combines videos to create new
synthetic videos with extended durations and enhanced resolutions, and
subsequently produces question-answer pairs pertaining to these newly
synthesized videos. Based on this paradigm, we develop seven video augmentation
methods and curate VISTA-400K, a video instruction-following dataset aimed at
enhancing long-duration and high-resolution video understanding. Finetuning
various video LMMs on our data resulted in an average improvement of 3.3%
across four challenging benchmarks for long-video understanding. Furthermore,
we introduce the first comprehensive high-resolution video understanding
benchmark HRVideoBench, on which our finetuned models achieve a 6.5%
performance gain. These results highlight the effectiveness of our framework.
674e99abe91289226e0428b4
Weiming Ren
claimed_verified
2024-12-03T11:14:38.615Z
64405a9d518271b0d1beea38
/avatars/b702474588fd7090773320422417a582.svg
Weiming Ren
user
wren93
674e99abe91289226e0428b5
Huan Yang
extracted_pending
2024-12-03T05:39:57.190Z
639005861c5a6237272557d1
https://cdn-avatars.huggingface.co/v1/production/uploads/1670382915881-noauth.jpeg
Huan Yang
user
hyang0511
674e99abe91289226e0428b6
Jie Min
claimed_verified
2024-12-05T09:28:29.622Z
6551419b35d81e515311780d
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/X0jRpQ3wEhLD6pfKlXIQe.jpeg
Jie Min
user
Stefan8
674e99abe91289226e0428b7
Cong Wei
claimed_verified
2024-12-05T09:28:27.503Z
64f8e358766ff9f3d2b0de84
https://cdn-avatars.huggingface.co/v1/production/uploads/64f8e358766ff9f3d2b0de84/R2P1YG-mRBh7TU9wkjGGk.jpeg
Cong Wei
user
CongWei1230
674e99abe91289226e0428b8
Wenhu Chen
admin_assigned
2024-12-03T15:22:40.605Z
6313a86154e6e5d9f0f94e04
https://cdn-avatars.huggingface.co/v1/production/uploads/1662232951344-6313a86154e6e5d9f0f94e04.jpeg
Wenhu Chen
user
wenhu
VISTA: Enhancing Long-Duration and High-Resolution Video Understanding
  by Video Spatiotemporal Augmentation
Current large multimodal models (LMMs) face significant challenges in
processing and comprehending long-duration or high-resolution videos, which is
mainly due to the lack of high-quality datasets. To address this issue from a
data-centric perspective, we propose VISTA, a simple yet effective Video
Spatiotemporal Augmentation framework that synthesizes long-duration and
high-resolution video instruction-following pairs from existing video-caption
datasets. VISTA spatially and temporally combines videos to create new
synthetic videos with extended durations and enhanced resolutions, and
subsequently produces question-answer pairs pertaining to these newly
synthesized videos. Based on this paradigm, we develop seven video augmentation
methods and curate VISTA-400K, a video instruction-following dataset aimed at
enhancing long-duration and high-resolution video understanding. Finetuning
various video LMMs on our data resulted in an average improvement of 3.3%
across four challenging benchmarks for long-video understanding. Furthermore,
we introduce the first comprehensive high-resolution video understanding
benchmark HRVideoBench, on which our finetuned models achieve a 6.5%
performance gain. These results highlight the effectiveness of our framework.
VISTA, a Video Spatiotemporal Augmentation framework, enhances long-duration and high-resolution video understanding through the creation of synthetic video datasets and finetuning of video LMMs.
multimodal models Video Spatiotemporal Augmentation video-caption datasets synthetic videos question-answer pairs video instruction-following dataset Finetuning long-video understanding high-resolution video understanding HRVideoBench