Probing the 3D Awareness of Visual Foundation Models
Recent advances in large-scale pretraining have yielded visual foundation
models with strong capabilities. Not only can recent models generalize to
arbitrary images for their training task, their intermediate representations
are useful for other visual tasks such as detection and segmentation. Given
that such models can classify, delineate, and localize objects in 2D, we ask
whether they also represent their 3D structure? In this work, we analyze the 3D
awareness of visual foundation models. We posit that 3D awareness implies that
representations (1) encode the 3D structure of the scene and (2) consistently
represent the surface across views. We conduct a series of experiments using
task-specific probes and zero-shot inference procedures on frozen features. Our
experiments reveal several limitations of the current models. Our code and
analysis can be found at https://github.com/mbanani/probe3d.
661c8718c153e4a0a2a83ad6
Mohamed El Banani
661c8718c153e4a0a2a83ad7
Amit Raj
661c8718c153e4a0a2a83ad8
Kevis-Kokitsi Maninis
661c8718c153e4a0a2a83ad9
Abhishek Kar
661c8718c153e4a0a2a83ada
Yuanzhen Li
admin_assigned
2024-04-18T11:43:50.210Z
6334ee5831a2be3938c85639
/avatars/9821af26a585f54b595ec39f061f2250.svg
Yuanzhen Li
user
yzli
661c8718c153e4a0a2a83adb
Michael Rubinstein
661c8718c153e4a0a2a83adc
Deqing Sun
661c8718c153e4a0a2a83add
Leonidas Guibas
661c8718c153e4a0a2a83ade
Justin Johnson
661c8718c153e4a0a2a83adf
Varun Jampani
admin_assigned
2024-04-18T11:43:31.148Z
630e7cfe3fc17ffc50f6e835
/avatars/0b742ff094a09f9374fafcd97ab9e002.svg
Varun Jampani
user
varunjampani
Probing the 3D Awareness of Visual Foundation Models
Recent advances in large-scale pretraining have yielded visual foundation
models with strong capabilities. Not only can recent models generalize to
arbitrary images for their training task, their intermediate representations
are useful for other visual tasks such as detection and segmentation. Given
that such models can classify, delineate, and localize objects in 2D, we ask
whether they also represent their 3D structure? In this work, we analyze the 3D
awareness of visual foundation models. We posit that 3D awareness implies that
representations (1) encode the 3D structure of the scene and (2) consistently
represent the surface across views. We conduct a series of experiments using
task-specific probes and zero-shot inference procedures on frozen features. Our
experiments reveal several limitations of the current models. Our code and
analysis can be found at https://github.com/mbanani/probe3d.
Visual foundation models are analyzed for their 3D awareness through task-specific probes and zero-shot inference on frozen features, revealing limitations.
3D awareness representations scene 3D structure surface views task-specific probes zero-shot inference