Robust Ego-Exo Correspondence with Long-Term Memory
Yijun Hu, Bing Fan, Xin Gu, 海青 任, Dongfang Liu, Heng Fan, Libo Zhang
Establishing object-level correspondence between egocentric and exocentric views is essential for AI assistants to deliver precise and intuitive visual guidance. However, this task presents numerous challenges, including extreme viewpoint variations, occlusions, and the presence of many small objects. Existing methods usually borrow solutions from video object segmentation, such as XSegTx and XView-XMem, which struggle to address aforementioned difficulties effectively. Recently, the Segment Anything Model 2 (SAM 2) has demonstrated strong generalization capabilities and shown excellent performance in video object segmentation. Nevertheless, SAM 2 faces significant challenges when directly applied to the ego-exo correspondence (EEC) task, including suboptimal feature fusion between views and inefficient memory management for long video sequences. To address these limitations, we propose a novel EEC framework based on SAM 2 with long-term memories by presenting a  dual-memory system and an adaptive Mixture-of-Experts module. Specifically, our approach features (1) a Memory-View Mixture-of-Experts module which consists of a dual-branch routing mechanism to adaptively assign contribution weights to each expert feature along both channel and spatial dimensions, and (2) a dual-memory bank system with a dedicated compression strategy to retain critical long-term information while eliminating redundancy. In the extensive experiments on the challenging EgoExo4D benchmark, our method, dubbed LM-EEC, achieves new state-of-the-art results and significantly outperforms existing methods and the SAM 2 baseline, showcasing its strong generalization across diverse scenarios. Our code will be released.