VideoLLM Knows When to Speak: Enhancing Time-Sensitive Video
  Comprehension with Video-Text Duet Interaction Format
Recent researches on video large language models (VideoLLM) predominantly
focus on model architectures and training datasets, leaving the interaction
format between the user and the model under-explored. In existing works, users
often interact with VideoLLMs by using the entire video and a query as input,
after which the model generates a response. This interaction format constrains
the application of VideoLLMs in scenarios such as live-streaming comprehension
where videos do not end and responses are required in a real-time manner, and
also results in unsatisfactory performance on time-sensitive tasks that
requires localizing video segments. In this paper, we focus on a video-text
duet interaction format. This interaction format is characterized by the
continuous playback of the video, and both the user and the model can insert
their text messages at any position during the video playback. When a text
message ends, the video continues to play, akin to the alternative of two
performers in a duet. We construct MMDuetIT, a video-text training dataset
designed to adapt VideoLLMs to video-text duet interaction format. We also
introduce the Multi-Answer Grounded Video Question Answering (MAGQA) task to
benchmark the real-time response ability of VideoLLMs. Trained on MMDuetIT,
MMDuet demonstrates that adopting the video-text duet interaction format
enables the model to achieve significant improvements in various time-sensitive
tasks (76% CIDEr on YouCook2 dense video captioning, 90\% mAP on QVHighlights
highlight detection and 25% R@0.5 on Charades-STA temporal video grounding)
with minimal training efforts, and also enable VideoLLMs to reply in a
real-time manner as the video plays. Code, data and demo are available at:
https://github.com/yellow-binary-tree/MMDuet.
6747ea29d59bc71f133e8d0a
Yueqian Wang
admin_assigned
2024-11-28T12:22:04.153Z
643672220b2c0d86d0b0f639
https://cdn-avatars.huggingface.co/v1/production/uploads/643672220b2c0d86d0b0f639/8nqZku8HxsSL_zR3XJPqK.png
Yueqian Wang
user
wangyueqian
6747ea29d59bc71f133e8d0b
Xiaojun Meng
claimed_verified
2025-05-26T08:44:57.246Z
676bd26f295f85d93ec5dc16
/avatars/b8c00e5b9730c735b9ff3af12ed921be.svg
Meng Jason
user
mengxj08
6747ea29d59bc71f133e8d0c
Yuxuan Wang
extracted_confirmed
2024-11-28T07:16:32.123Z
60b9e6837946aff342f734ae
/avatars/a711a6aa35757dfd7b78b26098a964fc.svg
Yuxuan Wang
user
ColorfulAI
6747ea29d59bc71f133e8d0d
Jianxin Liang
admin_assigned
2024-11-28T12:22:17.700Z
64d245829db917d3f0a256d2
/avatars/f7821992daa8d3a587b45b2ecbc4304e.svg
jianxin liang
user
ljx2000
6747ea29d59bc71f133e8d0e
Jiansheng Wei
admin_assigned
2024-11-28T12:22:30.128Z
6348ad7826a24bfaf4ae7924
/avatars/6af4ebc10cf6dffa18efc859144da841.svg
Wei JianSheng
user
jiansheng80
6747ea29d59bc71f133e8d0f
Huishuai Zhang
6747ea29d59bc71f133e8d10
Dongyan Zhao
admin_assigned
2024-11-28T12:22:53.269Z
66389190596ff1c3447cb88b
/avatars/e1eabfa6892810a830d8a42c1b5ee928.svg
zhao dongyang
user
yang2016
VideoLLM Knows When to Speak: Enhancing Time-Sensitive Video
  Comprehension with Video-Text Duet Interaction Format
Recent researches on video large language models (VideoLLM) predominantly
focus on model architectures and training datasets, leaving the interaction
format between the user and the model under-explored. In existing works, users
often interact with VideoLLMs by using the entire video and a query as input,
after which the model generates a response. This interaction format constrains
the application of VideoLLMs in scenarios such as live-streaming comprehension
where videos do not end and responses are required in a real-time manner, and
also results in unsatisfactory performance on time-sensitive tasks that
requires localizing video segments. In this paper, we focus on a video-text
duet interaction format. This interaction format is characterized by the
continuous playback of the video, and both the user and the model can insert
their text messages at any position during the video playback. When a text
message ends, the video continues to play, akin to the alternative of two
performers in a duet. We construct MMDuetIT, a video-text training dataset
designed to adapt VideoLLMs to video-text duet interaction format. We also
introduce the Multi-Answer Grounded Video Question Answering (MAGQA) task to
benchmark the real-time response ability of VideoLLMs. Trained on MMDuetIT,
MMDuet demonstrates that adopting the video-text duet interaction format
enables the model to achieve significant improvements in various time-sensitive
tasks (76% CIDEr on YouCook2 dense video captioning, 90\% mAP on QVHighlights
highlight detection and 25% R@0.5 on Charades-STA temporal video grounding)
with minimal training efforts, and also enable VideoLLMs to reply in a
real-time manner as the video plays. Code, data and demo are available at:
https://github.com/yellow-binary-tree/MMDuet.
VideoLLMs using a video-text duet interaction format achieve improvements in time-sensitive tasks and real-time response capabilities with minimal training effort.
video-text duet MMDuetIT Multi-Answer Grounded Video Question Answering (MAGQA) YouCook2 dense video captioning QVHighlights highlight detection Charades-STA temporal video grounding