3D Aware Region Prompted Vision Language Model
We present Spatial Region 3D (SR-3D) aware vision-language model that
connects single-view 2D images and multi-view 3D data through a shared visual
token space. SR-3D supports flexible region prompting, allowing users to
annotate regions with bounding boxes, segmentation masks on any frame, or
directly in 3D, without the need for exhaustive multi-frame labeling. We
achieve this by enriching 2D visual features with 3D positional embeddings,
which allows the 3D model to draw upon strong 2D priors for more accurate
spatial reasoning across frames, even when objects of interest do not co-occur
within the same view. Extensive experiments on both general 2D vision language
and specialized 3D spatial benchmarks demonstrate that SR-3D achieves
state-of-the-art performance, underscoring its effectiveness for unifying 2D
and 3D representation space on scene understanding. Moreover, we observe
applicability to in-the-wild videos without sensory 3D inputs or ground-truth
3D annotations, where SR-3D accurately infers spatial relationships and metric
measurements.
68ca17616e0073c09bd1de4b
An-Chieh Cheng
68ca17616e0073c09bd1de4c
Yang Fu
68ca17616e0073c09bd1de4d
Yukang Chen
68ca17616e0073c09bd1de4e
Zhijian Liu
68ca17616e0073c09bd1de4f
Xiaolong Li
68ca17616e0073c09bd1de50
Subhashree Radhakrishnan
68ca17616e0073c09bd1de51
Song Han
68ca17616e0073c09bd1de52
Yao Lu
68ca17616e0073c09bd1de53
Jan Kautz
68ca17616e0073c09bd1de54
Pavlo Molchanov
68ca17616e0073c09bd1de55
Hongxu Yin
68ca17616e0073c09bd1de56
Xiaolong Wang
68ca17616e0073c09bd1de57
Sifei Liu
3D Aware Region Prompted Vision Language Model
We present Spatial Region 3D (SR-3D) aware vision-language model that
connects single-view 2D images and multi-view 3D data through a shared visual
token space. SR-3D supports flexible region prompting, allowing users to
annotate regions with bounding boxes, segmentation masks on any frame, or
directly in 3D, without the need for exhaustive multi-frame labeling. We
achieve this by enriching 2D visual features with 3D positional embeddings,
which allows the 3D model to draw upon strong 2D priors for more accurate
spatial reasoning across frames, even when objects of interest do not co-occur
within the same view. Extensive experiments on both general 2D vision language
and specialized 3D spatial benchmarks demonstrate that SR-3D achieves
state-of-the-art performance, underscoring its effectiveness for unifying 2D
and 3D representation space on scene understanding. Moreover, we observe
applicability to in-the-wild videos without sensory 3D inputs or ground-truth
3D annotations, where SR-3D accurately infers spatial relationships and metric
measurements.
A Spatial Region 3D (SR-3D) vision-language model unifies 2D and 3D representations by enriching 2D features with 3D positional embeddings, enabling flexible region prompting and accurate spatial reasoning across frames.
Spatial Region 3D SR-3D vision-language model visual token space region prompting bounding boxes segmentation masks 3D positional embeddings spatial reasoning scene understanding in-the-wild videos spatial relationships metric measurements