VidEgoThink: Assessing Egocentric Video Understanding Capabilities for
  Embodied AI
Recent advancements in Multi-modal Large Language Models (MLLMs) have opened
new avenues for applications in Embodied AI. Building on previous work,
EgoThink, we introduce VidEgoThink, a comprehensive benchmark for evaluating
egocentric video understanding capabilities. To bridge the gap between MLLMs
and low-level control in Embodied AI, we design four key interrelated tasks:
video question-answering, hierarchy planning, visual grounding and reward
modeling. To minimize manual annotation costs, we develop an automatic data
generation pipeline based on the Ego4D dataset, leveraging the prior knowledge
and multimodal capabilities of GPT-4o. Three human annotators then filter the
generated data to ensure diversity and quality, resulting in the VidEgoThink
benchmark. We conduct extensive experiments with three types of models:
API-based MLLMs, open-source image-based MLLMs, and open-source video-based
MLLMs. Experimental results indicate that all MLLMs, including GPT-4o, perform
poorly across all tasks related to egocentric video understanding. These
findings suggest that foundation models still require significant advancements
to be effectively applied to first-person scenarios in Embodied AI. In
conclusion, VidEgoThink reflects a research trend towards employing MLLMs for
egocentric vision, akin to human capabilities, enabling active observation and
interaction in the complex real-world environments.
6710b440eac1980726739204
Sijie Cheng
claimed_verified
2024-10-17T11:24:13.175Z
639003391c5a623727252735
https://cdn-avatars.huggingface.co/v1/production/uploads/639003391c5a623727252735/fp_wZEhGzxGlyr64XF-x9.jpeg
Sijie Cheng
user
SijieCheng
6710b440eac1980726739205
Kechen Fang
6710b440eac1980726739206
Yangyang Yu
6710b440eac1980726739207
Sicheng Zhou
claimed_verified
2024-10-18T08:14:23.086Z
663c83439b8151660f29573e
/avatars/89b52fa42e198b79e4d5bd069c798d31.svg
Sicheng Zhou
user
eternity304
6710b440eac1980726739208
Bohao Li
claimed_verified
2025-03-28T08:39:08.694Z
6310b7e70a43f97f6c56191e
/avatars/4a24c76e34d12c3d6230a4a081115f72.svg
Bohao Li
user
BreakLee
6710b440eac1980726739209
Ye Tian
claimed_verified
2024-10-17T11:24:11.041Z
6414061f996b2e426f23df48
/avatars/db0af49c5f017f7c455a076b8b2212b6.svg
Ye Tian
user
yetian
6710b440eac198072673920a
Tingguang Li
6710b440eac198072673920b
Lei Han
6710b440eac198072673920c
Yang Liu
VidEgoThink: Assessing Egocentric Video Understanding Capabilities for
  Embodied AI
Recent advancements in Multi-modal Large Language Models (MLLMs) have opened
new avenues for applications in Embodied AI. Building on previous work,
EgoThink, we introduce VidEgoThink, a comprehensive benchmark for evaluating
egocentric video understanding capabilities. To bridge the gap between MLLMs
and low-level control in Embodied AI, we design four key interrelated tasks:
video question-answering, hierarchy planning, visual grounding and reward
modeling. To minimize manual annotation costs, we develop an automatic data
generation pipeline based on the Ego4D dataset, leveraging the prior knowledge
and multimodal capabilities of GPT-4o. Three human annotators then filter the
generated data to ensure diversity and quality, resulting in the VidEgoThink
benchmark. We conduct extensive experiments with three types of models:
API-based MLLMs, open-source image-based MLLMs, and open-source video-based
MLLMs. Experimental results indicate that all MLLMs, including GPT-4o, perform
poorly across all tasks related to egocentric video understanding. These
findings suggest that foundation models still require significant advancements
to be effectively applied to first-person scenarios in Embodied AI. In
conclusion, VidEgoThink reflects a research trend towards employing MLLMs for
egocentric vision, akin to human capabilities, enabling active observation and
interaction in the complex real-world environments.
VidEgoThink, a comprehensive benchmark for evaluating egocentric video understanding capabilities, highlights the need for advancements in Multi-modal Large Language Models for effective application in Embodied AI.
Multi-modal Large Language Models MLLMs Embodied AI VidEgoThink video question-answering hierarchy planning visual grounding reward modeling egocentric video understanding GPT-4o Ego4D dataset API-based MLLMs image-based MLLMs video-based MLLMs first-person scenarios