MMG-Ego4D: Multi-Modal Generalization in Egocentric Action Recognition
In this paper, we study a novel problem in egocentric action recognition,
which we term as "Multimodal Generalization" (MMG). MMG aims to study how
systems can generalize when data from certain modalities is limited or even
completely missing. We thoroughly investigate MMG in the context of standard
supervised action recognition and the more challenging few-shot setting for
learning new action categories. MMG consists of two novel scenarios, designed
to support security, and efficiency considerations in real-world applications:
(1) missing modality generalization where some modalities that were present
during the train time are missing during the inference time, and (2)
cross-modal zero-shot generalization, where the modalities present during the
inference time and the training time are disjoint. To enable this
investigation, we construct a new dataset MMG-Ego4D containing data points with
video, audio, and inertial motion sensor (IMU) modalities. Our dataset is
derived from Ego4D dataset, but processed and thoroughly re-annotated by human
experts to facilitate research in the MMG problem. We evaluate a diverse array
of models on MMG-Ego4D and propose new methods with improved generalization
ability. In particular, we introduce a new fusion module with modality dropout
training, contrastive-based alignment training, and a novel cross-modal
prototypical loss for better few-shot performance. We hope this study will
serve as a benchmark and guide future research in multimodal generalization
problems. The benchmark and code will be available at
https://github.com/facebookresearch/MMG_Ego4D.
64619c09846a6c8c8304b7aa
Xinyu Gong
claimed_verified
2023-05-15T15:28:10.203Z
63ab45de8a357d5057ab7ce5
/avatars/47c8cd29253ed8f1937c8223da30b80b.svg
Xinyu Gong
user
neogong
64619c09846a6c8c8304b7ab
Sreyas Mohan
admin_assigned
2023-06-02T12:53:54.656Z
63ad24a5073766d0850d01d8
/avatars/9d4662e639949c8ef7b431b09dd6d69d.svg
Sreyas Mohan
user
sreyasmohan
64619c09846a6c8c8304b7ac
Naina Dhingra
64619c09846a6c8c8304b7ad
Jean-Charles Bazin
64619c09846a6c8c8304b7ae
Yilei Li
64619c09846a6c8c8304b7af
Zhangyang Wang
64619c09846a6c8c8304b7b0
Rakesh Ranjan
MMG-Ego4D: Multi-Modal Generalization in Egocentric Action Recognition
In this paper, we study a novel problem in egocentric action recognition,
which we term as "Multimodal Generalization" (MMG). MMG aims to study how
systems can generalize when data from certain modalities is limited or even
completely missing. We thoroughly investigate MMG in the context of standard
supervised action recognition and the more challenging few-shot setting for
learning new action categories. MMG consists of two novel scenarios, designed
to support security, and efficiency considerations in real-world applications:
(1) missing modality generalization where some modalities that were present
during the train time are missing during the inference time, and (2)
cross-modal zero-shot generalization, where the modalities present during the
inference time and the training time are disjoint. To enable this
investigation, we construct a new dataset MMG-Ego4D containing data points with
video, audio, and inertial motion sensor (IMU) modalities. Our dataset is
derived from Ego4D dataset, but processed and thoroughly re-annotated by human
experts to facilitate research in the MMG problem. We evaluate a diverse array
of models on MMG-Ego4D and propose new methods with improved generalization
ability. In particular, we introduce a new fusion module with modality dropout
training, contrastive-based alignment training, and a novel cross-modal
prototypical loss for better few-shot performance. We hope this study will
serve as a benchmark and guide future research in multimodal generalization
problems. The benchmark and code will be available at
https://github.com/facebookresearch/MMG_Ego4D.
Research investigates multimodal generalization in egocentric action recognition through two scenarios: missing and cross-modal zero-shot, using a new dataset with video, audio, and IMU data, and introduces a fusion module, contrastive alignment, and cross-modal prototypical loss for improved few-shot performance.
multimodal generalization egocentric action recognition missing modality generalization cross-modal zero-shot generalization MMG-Ego4D modality dropout contrastive-based alignment cross-modal prototypical loss few-shot learning