C3Po: Cross-View Cross-Modality Correspondence by Pointmap Prediction
Kuan Wei Huang, Brandon Li, Bharath Hariharan, Noah Snavely
Geometric models like DUSt3R have shown great advances in understanding the geometry of a scene from pairs of photos. However, they fail when the inputs are from vastly different viewpoints (e.g., aerial vs. ground) or modalities (e.g., photos vs. abstract drawings) due to the vast differences in viewpoint or style compared to what was observed during training. This paper addresses a challenging version of this problem: predicting correspondences between ground-level photos and floor plans. Current datasets for joint photo--floor plan reasoning are limited, either lacking in varying modalities (VIGOR) or lacking in correspondences (WAFFLE). To address these limitations, we introduce a new dataset, C3, created by first reconstructing a number of scenes in 3D from Internet photo collections via structure from motion, then manually registering the reconstructions to floor plans gathered from the Internet, from which we can derive correspondence between images and floor plans. C3 contains 91K paired floor plans and photos across 574 scenes with 155M pixel-level correspondences. We find that state-of-the-art correspondence models struggle on this task. By training on our new data, we can improve on the best performing method by 34\% in RMSE. However, we also identify open challenges in cross-modal geometric reasoning that our dataset aims to help address.