CrossOver: 3D Scene Cross-Modal Alignment
Multi-modal 3D object understanding has gained significant attention, yet
current approaches often assume complete data availability and rigid alignment
across all modalities. We present CrossOver, a novel framework for cross-modal
3D scene understanding via flexible, scene-level modality alignment. Unlike
traditional methods that require aligned modality data for every object
instance, CrossOver learns a unified, modality-agnostic embedding space for
scenes by aligning modalities - RGB images, point clouds, CAD models,
floorplans, and text descriptions - with relaxed constraints and without
explicit object semantics. Leveraging dimensionality-specific encoders, a
multi-stage training pipeline, and emergent cross-modal behaviors, CrossOver
supports robust scene retrieval and object localization, even with missing
modalities. Evaluations on ScanNet and 3RScan datasets show its superior
performance across diverse metrics, highlighting adaptability for real-world
applications in 3D scene understanding.
67bc0d12ffc2c387329c8cfd
Sayan Deb Sarkar
claimed_verified
2025-02-24T09:06:56.555Z
650ec19e6620b0c57e2a551b
/avatars/c26c03fa920d857120f03c9ccb9f1d7a.svg
Sayan Deb Sarkar
user
sayandsarkar
67bc0d12ffc2c387329c8cfe
Ondrej Miksik
67bc0d12ffc2c387329c8cff
Marc Pollefeys
admin_assigned
2025-02-24T16:11:04.481Z
67b5fa179782a5e2fd2cb26a
/avatars/62c38f29ec641e001eeddf840bea21a0.svg
Marc Pollefeys
user
mapo1
67bc0d12ffc2c387329c8d00
Daniel Barath
67bc0d12ffc2c387329c8d01
Iro Armeni
admin_assigned
2025-02-24T16:11:14.554Z
6745f90cf4d75fd11a2407ac
/avatars/882f56565b4ebfabf1c13e199d74a4de.svg
Iro Armeni
user
ir0
CrossOver: 3D Scene Cross-Modal Alignment
Multi-modal 3D object understanding has gained significant attention, yet
current approaches often assume complete data availability and rigid alignment
across all modalities. We present CrossOver, a novel framework for cross-modal
3D scene understanding via flexible, scene-level modality alignment. Unlike
traditional methods that require aligned modality data for every object
instance, CrossOver learns a unified, modality-agnostic embedding space for
scenes by aligning modalities - RGB images, point clouds, CAD models,
floorplans, and text descriptions - with relaxed constraints and without
explicit object semantics. Leveraging dimensionality-specific encoders, a
multi-stage training pipeline, and emergent cross-modal behaviors, CrossOver
supports robust scene retrieval and object localization, even with missing
modalities. Evaluations on ScanNet and 3RScan datasets show its superior
performance across diverse metrics, highlighting adaptability for real-world
applications in 3D scene understanding.
CrossOver, a framework for cross-modal 3D scene understanding, achieves robust scene retrieval and object localization by learning a unified embedding space across multiple modalities with relaxed constraints and adaptability to missing modalities.
CrossOver cross-modal 3D scene understanding unified embedding space dimensionality-specific encoders multi-stage training pipeline emergent cross-modal behaviors ScanNet 3RScan