ViSpeak: Visual Instruction Feedback in Streaming Videos
Recent advances in Large Multi-modal Models (LMMs) are primarily focused on
offline video understanding. Instead, streaming video understanding poses great
challenges to recent models due to its time-sensitive, omni-modal and
interactive characteristics. In this work, we aim to extend the streaming video
understanding from a new perspective and propose a novel task named Visual
Instruction Feedback in which models should be aware of visual contents and
learn to extract instructions from them. For example, when users wave their
hands to agents, agents should recognize the gesture and start conversations
with welcome information. Thus, following instructions in visual modality
greatly enhances user-agent interactions. To facilitate research, we define
seven key subtasks highly relevant to visual modality and collect the
ViSpeak-Instruct dataset for training and the ViSpeak-Bench for evaluation.
Further, we propose the ViSpeak model, which is a SOTA streaming video
understanding LMM with GPT-4o-level performance on various streaming video
understanding benchmarks. After finetuning on our ViSpeak-Instruct dataset,
ViSpeak is equipped with basic visual instruction feedback ability, serving as
a solid baseline for future research.
67d8ded81a1b6ae91f79eb18
Shenghao Fu
claimed_verified
2025-03-20T10:46:26.291Z
67067633351e0c16a5c27497
/avatars/356aa3431198c8931b820a714bcfb19d.svg
Shenghao Fu
user
fushh7
67d8ded81a1b6ae91f79eb19
Qize Yang
claimed_verified
2025-03-20T10:46:23.971Z
66a097801a26a2350395edc7
/avatars/1e7e127cb7222df7d56e5bfda6bab519.svg
Qize Yang
user
PhilipC
67d8ded81a1b6ae91f79eb1a
Yuan-Ming Li
claimed_verified
2025-03-20T10:46:20.167Z
644fe6a9e1d7a97f3b66e906
/avatars/ad1a45f0b1c8a4d03ba87f2a3ce5a8f8.svg
Yuanming-Li
user
Lymann
67d8ded81a1b6ae91f79eb1b
Yi-Xing Peng
claimed_verified
2025-03-20T10:46:18.304Z
66b02f7405e2b2771bb431db
/avatars/9d1ecb38e6cb2f0be33a5c1938bb1253.svg
Yi-Xing Peng
user
maybetomorrow
67d8ded81a1b6ae91f79eb1c
Kun-Yu Lin
67d8ded81a1b6ae91f79eb1d
Xihan Wei
67d8ded81a1b6ae91f79eb1e
Jian-Fang Hu
67d8ded81a1b6ae91f79eb1f
Xiaohua Xie
67d8ded81a1b6ae91f79eb20
Wei-Shi Zheng
ViSpeak: Visual Instruction Feedback in Streaming Videos
Recent advances in Large Multi-modal Models (LMMs) are primarily focused on
offline video understanding. Instead, streaming video understanding poses great
challenges to recent models due to its time-sensitive, omni-modal and
interactive characteristics. In this work, we aim to extend the streaming video
understanding from a new perspective and propose a novel task named Visual
Instruction Feedback in which models should be aware of visual contents and
learn to extract instructions from them. For example, when users wave their
hands to agents, agents should recognize the gesture and start conversations
with welcome information. Thus, following instructions in visual modality
greatly enhances user-agent interactions. To facilitate research, we define
seven key subtasks highly relevant to visual modality and collect the
ViSpeak-Instruct dataset for training and the ViSpeak-Bench for evaluation.
Further, we propose the ViSpeak model, which is a SOTA streaming video
understanding LMM with GPT-4o-level performance on various streaming video
understanding benchmarks. After finetuning on our ViSpeak-Instruct dataset,
ViSpeak is equipped with basic visual instruction feedback ability, serving as
a solid baseline for future research.
A novel Visual Instruction Feedback task is introduced for streaming video understanding, using the ViSpeak model with GPT-4o-level performance on relevant benchmarks.
Large Multi-modal Models streaming video understanding visual instruction feedback ViSpeak-Instruct dataset ViSpeak-Bench ViSpeak model