A Coarse-to-Fine Approach to Multi-Modality 3D Occupancy Grounding
Visual grounding aims to identify objects or regions in a scene based on
natural language descriptions, essential for spatially aware perception in
autonomous driving. However, existing visual grounding tasks typically depend
on bounding boxes that often fail to capture fine-grained details. Not all
voxels within a bounding box are occupied, resulting in inaccurate object
representations. To address this, we introduce a benchmark for 3D occupancy
grounding in challenging outdoor scenes. Built on the nuScenes dataset, it
integrates natural language with voxel-level occupancy annotations, offering
more precise object perception compared to the traditional grounding task.
Moreover, we propose GroundingOcc, an end-to-end model designed for 3D
occupancy grounding through multi-modal learning. It combines visual, textual,
and point cloud features to predict object location and occupancy information
from coarse to fine. Specifically, GroundingOcc comprises a multimodal encoder
for feature extraction, an occupancy head for voxel-wise predictions, and a
grounding head to refine localization. Additionally, a 2D grounding module and
a depth estimation module enhance geometric understanding, thereby boosting
model performance. Extensive experiments on the benchmark demonstrate that our
method outperforms existing baselines on 3D occupancy grounding. The dataset is
available at https://github.com/RONINGOD/GroundingOcc.
6894158a741a16f544fbcedd
Zhan Shi
6894158a741a16f544fbcede
Song Wang
claimed_verified
2025-08-07T10:38:41.313Z
66863d26e2b71e3d09189ae9
/avatars/3c0e6f30e053f2e622ae75e1dc43edba.svg
Song Wang
user
songw-zju
6894158a741a16f544fbcedf
Junbo Chen
6894158a741a16f544fbcee0
Jianke Zhu
A Coarse-to-Fine Approach to Multi-Modality 3D Occupancy Grounding
Visual grounding aims to identify objects or regions in a scene based on
natural language descriptions, essential for spatially aware perception in
autonomous driving. However, existing visual grounding tasks typically depend
on bounding boxes that often fail to capture fine-grained details. Not all
voxels within a bounding box are occupied, resulting in inaccurate object
representations. To address this, we introduce a benchmark for 3D occupancy
grounding in challenging outdoor scenes. Built on the nuScenes dataset, it
integrates natural language with voxel-level occupancy annotations, offering
more precise object perception compared to the traditional grounding task.
Moreover, we propose GroundingOcc, an end-to-end model designed for 3D
occupancy grounding through multi-modal learning. It combines visual, textual,
and point cloud features to predict object location and occupancy information
from coarse to fine. Specifically, GroundingOcc comprises a multimodal encoder
for feature extraction, an occupancy head for voxel-wise predictions, and a
grounding head to refine localization. Additionally, a 2D grounding module and
a depth estimation module enhance geometric understanding, thereby boosting
model performance. Extensive experiments on the benchmark demonstrate that our
method outperforms existing baselines on 3D occupancy grounding. The dataset is
available at https://github.com/RONINGOD/GroundingOcc.
A benchmark and model for 3D occupancy grounding using natural language and voxel-level annotations improve object perception in autonomous driving.
visual grounding 3D occupancy grounding nuScenes dataset voxel-level occupancy multimodal learning multimodal encoder occupancy head grounding head 2D grounding module depth estimation module