Video Instruction Tuning With Synthetic Data
The development of video large multimodal models (LMMs) has been hindered by
the difficulty of curating large amounts of high-quality raw data from the web.
To address this, we propose an alternative approach by creating a high-quality
synthetic dataset specifically for video instruction-following, namely
LLaVA-Video-178K. This dataset includes key tasks such as detailed captioning,
open-ended question-answering (QA), and multiple-choice QA. By training on this
dataset, in combination with existing visual instruction tuning data, we
introduce LLaVA-Video, a new video LMM. Our experiments demonstrate that
LLaVA-Video achieves strong performance across various video benchmarks,
highlighting the effectiveness of our dataset. We plan to release the dataset,
its generation pipeline, and the model checkpoints.
66ff4f5e5a359c1af1169664
Yuanhan Zhang
admin_assigned
2024-10-04T08:39:48.736Z
62a993d80472c0b7f94027df
https://cdn-avatars.huggingface.co/v1/production/uploads/62a993d80472c0b7f94027df/j5vp-IwLA2YBexylUHiQU.png
Zhang Yuanhan
user
ZhangYuanhan
66ff4f5e5a359c1af1169665
Jinming Wu
admin_assigned
2024-10-04T08:40:01.301Z
652fbe8cb2acab0b82f855a6
https://cdn-avatars.huggingface.co/v1/production/uploads/652fbe8cb2acab0b82f855a6/lVpzeEoFRQ6dnGAoNS9b3.jpeg
Jinming Wu
user
kimingng
66ff4f5e5a359c1af1169666
Wei Li
66ff4f5e5a359c1af1169667
Bo Li
claimed_verified
2024-10-21T07:48:56.361Z
62d3f7d84b0933c48f3cdd9c
https://cdn-avatars.huggingface.co/v1/production/uploads/62d3f7d84b0933c48f3cdd9c/rSoSyH0Td9fD9HVlyK7bh.jpeg
Bo Li
user
luodian
66ff4f5e5a359c1af1169668
Zejun Ma
66ff4f5e5a359c1af1169669
Ziwei Liu
admin_assigned
2024-10-04T08:40:14.432Z
62ab1ac1d48b4d8b048a3473
https://cdn-avatars.huggingface.co/v1/production/uploads/1656826685333-62ab1ac1d48b4d8b048a3473.png
Ziwei Liu
user
liuziwei7
66ff4f5e5a359c1af116966a
Chunyuan Li
admin_assigned
2024-10-04T08:40:08.501Z
62aba526cae4462c0c6caa0f
/avatars/430560ec2c2547f819225769ab432f30.svg
Chunyuan Li
user
Chunyuan24
Video Instruction Tuning With Synthetic Data
The development of video large multimodal models (LMMs) has been hindered by
the difficulty of curating large amounts of high-quality raw data from the web.
To address this, we propose an alternative approach by creating a high-quality
synthetic dataset specifically for video instruction-following, namely
LLaVA-Video-178K. This dataset includes key tasks such as detailed captioning,
open-ended question-answering (QA), and multiple-choice QA. By training on this
dataset, in combination with existing visual instruction tuning data, we
introduce LLaVA-Video, a new video LMM. Our experiments demonstrate that
LLaVA-Video achieves strong performance across various video benchmarks,
highlighting the effectiveness of our dataset. We plan to release the dataset,
its generation pipeline, and the model checkpoints.
A synthetic dataset, LLaVA-Video-178K, and a new video large multimodal model, LLaVA-Video, achieve strong performance in video instruction-following tasks.
video large multimodal models synthetic dataset LLaVA-Video-178K detailed captioning open-ended question-answering multiple-choice QA visual instruction tuning video benchmarks