OmniMMI: A Comprehensive Multi-modal Interaction Benchmark in Streaming
  Video Contexts
The rapid advancement of multi-modal language models (MLLMs) like GPT-4o has
propelled the development of Omni language models, designed to process and
proactively respond to continuous streams of multi-modal data. Despite their
potential, evaluating their real-world interactive capabilities in streaming
video contexts remains a formidable challenge. In this work, we introduce
OmniMMI, a comprehensive multi-modal interaction benchmark tailored for
OmniLLMs in streaming video contexts. OmniMMI encompasses over 1,121 videos and
2,290 questions, addressing two critical yet underexplored challenges in
existing video benchmarks: streaming video understanding and proactive
reasoning, across six distinct subtasks. Moreover, we propose a novel
framework, Multi-modal Multiplexing Modeling (M4), designed to enable an
inference-efficient streaming model that can see, listen while generating.
67eb58e5969b278277adb831
Yuxuan Wang
claimed_verified
2025-04-02T09:42:25.151Z
60b9e6837946aff342f734ae
/avatars/a711a6aa35757dfd7b78b26098a964fc.svg
Yuxuan Wang
user
ColorfulAI
67eb58e5969b278277adb832
Yueqian Wang
67eb58e5969b278277adb833
Bo Chen
67eb58e5969b278277adb834
Tong Wu
67eb58e5969b278277adb835
Dongyan Zhao
67eb58e5969b278277adb836
Zilong Zheng
extracted_pending
2025-04-01T03:09:26.871Z
63a95a6a7930fa8c7dd63d4e
/avatars/d9d0420f7ddfe2f3a7e029fb05f1c89f.svg
Zilong Zheng
user
zlzheng
OmniMMI: A Comprehensive Multi-modal Interaction Benchmark in Streaming
  Video Contexts
The rapid advancement of multi-modal language models (MLLMs) like GPT-4o has
propelled the development of Omni language models, designed to process and
proactively respond to continuous streams of multi-modal data. Despite their
potential, evaluating their real-world interactive capabilities in streaming
video contexts remains a formidable challenge. In this work, we introduce
OmniMMI, a comprehensive multi-modal interaction benchmark tailored for
OmniLLMs in streaming video contexts. OmniMMI encompasses over 1,121 videos and
2,290 questions, addressing two critical yet underexplored challenges in
existing video benchmarks: streaming video understanding and proactive
reasoning, across six distinct subtasks. Moreover, we propose a novel
framework, Multi-modal Multiplexing Modeling (M4), designed to enable an
inference-efficient streaming model that can see, listen while generating.
OmniMMI is a benchmark and M4 is a framework aimed at evaluating and enabling inference-efficient multi-modal processing and proactive reasoning in streaming video contexts for OmniLLMs.
multi-modal language models Omni language models multi-modal interaction benchmark streaming video understanding proactive reasoning Multi-modal Multiplexing Modeling inference-efficient streaming model