Vript: A Video Is Worth Thousands of Words
Advancements in multimodal learning, particularly in video understanding and
generation, require high-quality video-text datasets for improved model
performance. Vript addresses this issue with a meticulously annotated corpus of
12K high-resolution videos, offering detailed, dense, and script-like captions
for over 420K clips. Each clip has a caption of ~145 words, which is over 10x
longer than most video-text datasets. Unlike captions only documenting static
content in previous datasets, we enhance video captioning to video scripting by
documenting not just the content, but also the camera operations, which include
the shot types (medium shot, close-up, etc) and camera movements (panning,
tilting, etc). By utilizing the Vript, we explore three training paradigms of
aligning more text with the video modality rather than clip-caption pairs. This
results in Vriptor, a top-performing video captioning model among open-source
models, comparable to GPT-4V in performance. Vriptor is also a powerful model
capable of end-to-end generation of dense and detailed captions for long
videos. Moreover, we introduce Vript-Hard, a benchmark consisting of three
video understanding tasks that are more challenging than existing benchmarks:
Vript-HAL is the first benchmark evaluating action and object hallucinations in
video LLMs, Vript-RR combines reasoning with retrieval resolving question
ambiguity in long-video QAs, and Vript-ERO is a new task to evaluate the
temporal understanding of events in long videos rather than actions in short
videos in previous works. All code, models, and datasets are available in
https://github.com/mutonix/Vript.
6667ce1bf3e4efa2764a9f36
Dongjie Yang
claimed_verified
2024-06-11T07:55:45.022Z
63423e6206a304bdc43866bd
https://cdn-avatars.huggingface.co/v1/production/uploads/63423e6206a304bdc43866bd/NZ4Oo9APjqxmnhI4tzk55.jpeg
Dongjie Yang
user
Mutonix
6667ce1bf3e4efa2764a9f37
Suyuan Huang
admin_assigned
2024-06-11T07:58:57.106Z
645de4648c179ee09e14e1b1
/avatars/1e90a93f251e35c16ec6de15391a3e9e.svg
Suyuan Huang
user
Florient
6667ce1bf3e4efa2764a9f38
Chengqiang Lu
claimed_verified
2025-02-21T14:43:32.869Z
61de751fc59d602a3a85b1aa
https://cdn-avatars.huggingface.co/v1/production/uploads/1641968925128-noauth.jpeg
lu
user
cq
6667ce1bf3e4efa2764a9f39
Xiaodong Han
admin_assigned
2024-06-11T07:59:10.593Z
64638bddc615cbc12447f9f1
/avatars/52910d1717451ff983f745322e5850dd.svg
Xiaodong Han
user
Hannnnnxd
6667ce1bf3e4efa2764a9f3a
Haoxin Zhang
6667ce1bf3e4efa2764a9f3b
Yan Gao
6667ce1bf3e4efa2764a9f3c
Yao Hu
6667ce1bf3e4efa2764a9f3d
Hai Zhao
Vript: A Video Is Worth Thousands of Words
Advancements in multimodal learning, particularly in video understanding and
generation, require high-quality video-text datasets for improved model
performance. Vript addresses this issue with a meticulously annotated corpus of
12K high-resolution videos, offering detailed, dense, and script-like captions
for over 420K clips. Each clip has a caption of ~145 words, which is over 10x
longer than most video-text datasets. Unlike captions only documenting static
content in previous datasets, we enhance video captioning to video scripting by
documenting not just the content, but also the camera operations, which include
the shot types (medium shot, close-up, etc) and camera movements (panning,
tilting, etc). By utilizing the Vript, we explore three training paradigms of
aligning more text with the video modality rather than clip-caption pairs. This
results in Vriptor, a top-performing video captioning model among open-source
models, comparable to GPT-4V in performance. Vriptor is also a powerful model
capable of end-to-end generation of dense and detailed captions for long
videos. Moreover, we introduce Vript-Hard, a benchmark consisting of three
video understanding tasks that are more challenging than existing benchmarks:
Vript-HAL is the first benchmark evaluating action and object hallucinations in
video LLMs, Vript-RR combines reasoning with retrieval resolving question
ambiguity in long-video QAs, and Vript-ERO is a new task to evaluate the
temporal understanding of events in long videos rather than actions in short
videos in previous works. All code, models, and datasets are available in
https://github.com/mutonix/Vript.
Vript introduces a high-quality video-text dataset with detailed captions including camera operations, enhancing video captioning and generation, and proposes Vriptor, a top-performing model with Vript-Hard, a new benchmark for video understanding challenges.
multimodal learning video understanding video generation Vript high-resolution videos dense captions script-like captions shot types camera movements Vriptor GPT-4V video captioning Vript-Hard Vript-HAL Vript-RR Vript-ERO temporal understanding long-video QAs action and object hallucinations