Escaping Plato's Cave: Towards the Alignment of 3D and Text Latent
  Spaces
Recent works have shown that, when trained at scale, uni-modal 2D vision and
text encoders converge to learned features that share remarkable structural
properties, despite arising from different representations. However, the role
of 3D encoders with respect to other modalities remains unexplored.
Furthermore, existing 3D foundation models that leverage large datasets are
typically trained with explicit alignment objectives with respect to frozen
encoders from other representations. In this work, we investigate the
possibility of a posteriori alignment of representations obtained from
uni-modal 3D encoders compared to text-based feature spaces. We show that naive
post-training feature alignment of uni-modal text and 3D encoders results in
limited performance. We then focus on extracting subspaces of the corresponding
feature spaces and discover that by projecting learned representations onto
well-chosen lower-dimensional subspaces the quality of alignment becomes
significantly higher, leading to improved accuracy on matching and retrieval
tasks. Our analysis further sheds light on the nature of these shared
subspaces, which roughly separate between semantic and geometric data
representations. Overall, ours is the first work that helps to establish a
baseline for post-training alignment of 3D uni-modal and text feature spaces,
and helps to highlight both the shared and unique properties of 3D data
compared to other representations.
67cef721e5ab8ec0550b7a66
Souhail Hadgi
admin_assigned
2025-03-11T14:18:09.155Z
6542cb9602409f0994fbb6c6
/avatars/c0ba12b349e61998f36f2701ac3acb30.svg
Souhail Hadgi
user
Squall01
67cef721e5ab8ec0550b7a67
Luca Moschella
admin_assigned
2025-03-11T14:18:16.141Z
607021f450a44430c625a937
https://cdn-avatars.huggingface.co/v1/production/uploads/1617961457504-noauth.jpeg
Luca Moschella
user
lucmos
67cef721e5ab8ec0550b7a68
Andrea Santilli
claimed_verified
2025-03-10T14:35:44.397Z
5e8ef1f14957053f606489e6
https://cdn-avatars.huggingface.co/v1/production/uploads/1635502086699-5e8ef1f14957053f606489e6.jpeg
Andrea Santilli
user
teelinsan
67cef721e5ab8ec0550b7a69
Diego Gomez
67cef721e5ab8ec0550b7a6a
Qixing Huang
claimed_verified
2025-04-08T06:58:01.534Z
67f18d051358145ee5bb3a11
/avatars/e5a8c12295d7609625c2b891d47c3112.svg
qixing huang
user
qixinghuang1
67cef721e5ab8ec0550b7a6b
Emanuele Rodolà
admin_assigned
2025-03-11T14:18:47.736Z
652681664e066bf73f8e2bd1
/avatars/084dec4765d9996d74901b8df95ec35f.svg
Emanuele Rodola'
user
erodola
67cef721e5ab8ec0550b7a6c
Simone Melzi
67cef721e5ab8ec0550b7a6d
Maks Ovsjanikov
Escaping Plato's Cave: Towards the Alignment of 3D and Text Latent
  Spaces
Recent works have shown that, when trained at scale, uni-modal 2D vision and
text encoders converge to learned features that share remarkable structural
properties, despite arising from different representations. However, the role
of 3D encoders with respect to other modalities remains unexplored.
Furthermore, existing 3D foundation models that leverage large datasets are
typically trained with explicit alignment objectives with respect to frozen
encoders from other representations. In this work, we investigate the
possibility of a posteriori alignment of representations obtained from
uni-modal 3D encoders compared to text-based feature spaces. We show that naive
post-training feature alignment of uni-modal text and 3D encoders results in
limited performance. We then focus on extracting subspaces of the corresponding
feature spaces and discover that by projecting learned representations onto
well-chosen lower-dimensional subspaces the quality of alignment becomes
significantly higher, leading to improved accuracy on matching and retrieval
tasks. Our analysis further sheds light on the nature of these shared
subspaces, which roughly separate between semantic and geometric data
representations. Overall, ours is the first work that helps to establish a
baseline for post-training alignment of 3D uni-modal and text feature spaces,
and helps to highlight both the shared and unique properties of 3D data
compared to other representations.
Post-training alignment of 3D and text feature spaces improves accuracy by projecting representations onto optimal subspaces, revealing shared semantic and geometric characteristics.
uni-modal 2D vision text encoders 3D encoders 3D foundation models explicit alignment objectives frozen encoders post-training feature alignment feature spaces lower-dimensional subspaces semantic data representations geometric data representations