MM-Ego: Towards Building Egocentric Multimodal LLMs
This research aims to comprehensively explore building a multimodal
foundation model for egocentric video understanding. To achieve this goal, we
work on three fronts. First, as there is a lack of QA data for egocentric video
understanding, we develop a data engine that efficiently generates 7M
high-quality QA samples for egocentric videos ranging from 30 seconds to one
hour long, based on human-annotated data. This is currently the largest
egocentric QA dataset. Second, we contribute a challenging egocentric QA
benchmark with 629 videos and 7,026 questions to evaluate the models' ability
in recognizing and memorizing visual details across videos of varying lengths.
We introduce a new de-biasing evaluation method to help mitigate the
unavoidable language bias present in the models being evaluated. Third, we
propose a specialized multimodal architecture featuring a novel "Memory Pointer
Prompting" mechanism. This design includes a global glimpse step to gain an
overarching understanding of the entire video and identify key visual
information, followed by a fallback step that utilizes the key visual
information to generate responses. This enables the model to more effectively
comprehend extended video content. With the data, benchmark, and model, we
successfully build MM-Ego, an egocentric multimodal LLM that shows powerful
performance on egocentric video understanding.
67074fe8784e1c990108328a
Hanrong Ye
admin_assigned
2024-10-10T08:56:29.882Z
62d3ae4d894e7fe42def988f
/avatars/3aafc55d9783459f9a79546fc31dd68a.svg
Hanrong Ye
user
leoye
67074fe8784e1c990108328b
Haotian Zhang
claimed_verified
2024-10-10T08:00:37.565Z
631516348d85ad332fa47b2c
/avatars/100f5ae3cf3c52faaecdaecd5d8f2881.svg
Haotian Zhang
user
haotiz
67074fe8784e1c990108328c
Erik Daxberger
admin_assigned
2024-10-10T08:56:45.821Z
66150e514476535462a7166c
/avatars/75a0b6546dde9853a06c9536f067005d.svg
Erik Daxberger
user
edaxberger
67074fe8784e1c990108328d
Lin Chen
claimed_verified
2025-06-05T10:00:37.064Z
67573b050fef6dcc436cd182
/avatars/9e77f4de66b924ece97a8c2a38d46531.svg
Lin Chen
user
chenlzk
67074fe8784e1c990108328e
Zongyu Lin
67074fe8784e1c990108328f
Yanghao Li
admin_assigned
2024-10-10T08:58:00.320Z
65dad3870af7e21ba473439f
/avatars/da542e7d68ae937bbdb791f17096bb1c.svg
Yanghao Li
user
FrozzZen
67074fe8784e1c9901083290
Bowen Zhang
67074fe8784e1c9901083291
Haoxuan You
admin_assigned
2024-10-10T08:57:36.252Z
65284ba3e3b3844f5e487a0b
/avatars/fce12a233bddb611d441f25a2ab17725.svg
Haoxuan You
user
HaoxuanYou
67074fe8784e1c9901083292
Dan Xu
admin_assigned
2024-10-10T08:57:29.789Z
66feab48651e00e22f33222e
/avatars/7344377e2c796c7ec85194bb2fc78521.svg
Dan Xu
user
danxuhk
67074fe8784e1c9901083293
Zhe Gan
admin_assigned
2024-10-10T08:57:03.855Z
644022db21ec7c165d848b6e
/avatars/d9ed37ec77f2682878b2976a04d2fcd8.svg
Zhe Gan
user
zhegan27
67074fe8784e1c9901083294
Jiasen Lu
admin_assigned
2024-10-10T08:56:57.823Z
62b6b0397523238923221df9
/avatars/77068771dd51df7519516cd502a88789.svg
Jiasenlu
user
Jiasenlu
67074fe8784e1c9901083295
Yinfei Yang
admin_assigned
2024-10-10T08:56:52.265Z
64b762568c632fbca942a405
/avatars/1eb737ec169967872f1ebf5ff29f1e6b.svg
Yinfei Yang
user
yinfeiy
MM-Ego: Towards Building Egocentric Multimodal LLMs
This research aims to comprehensively explore building a multimodal
foundation model for egocentric video understanding. To achieve this goal, we
work on three fronts. First, as there is a lack of QA data for egocentric video
understanding, we develop a data engine that efficiently generates 7M
high-quality QA samples for egocentric videos ranging from 30 seconds to one
hour long, based on human-annotated data. This is currently the largest
egocentric QA dataset. Second, we contribute a challenging egocentric QA
benchmark with 629 videos and 7,026 questions to evaluate the models' ability
in recognizing and memorizing visual details across videos of varying lengths.
We introduce a new de-biasing evaluation method to help mitigate the
unavoidable language bias present in the models being evaluated. Third, we
propose a specialized multimodal architecture featuring a novel "Memory Pointer
Prompting" mechanism. This design includes a global glimpse step to gain an
overarching understanding of the entire video and identify key visual
information, followed by a fallback step that utilizes the key visual
information to generate responses. This enables the model to more effectively
comprehend extended video content. With the data, benchmark, and model, we
successfully build MM-Ego, an egocentric multimodal LLM that shows powerful
performance on egocentric video understanding.
A multimodal foundation model for egocentric video understanding is developed with a large QA dataset, a challenging benchmark, and a specialized architecture featuring a "Memory Pointer Prompting" mechanism.
QA data egocentric video understanding data engine QA samples de-biasing evaluation method multimodal architecture Memory Pointer Prompting global glimpse step fallback step multimodal LLM MM-Ego