SyntheOcc: Synthesize Geometric-Controlled Street View Images through 3D
  Semantic MPIs
The advancement of autonomous driving is increasingly reliant on high-quality
annotated datasets, especially in the task of 3D occupancy prediction, where
the occupancy labels require dense 3D annotation with significant human effort.
In this paper, we propose SyntheOcc, which denotes a diffusion model that
Synthesize photorealistic and geometric-controlled images by conditioning
Occupancy labels in driving scenarios. This yields an unlimited amount of
diverse, annotated, and controllable datasets for applications like training
perception models and simulation. SyntheOcc addresses the critical challenge of
how to efficiently encode 3D geometric information as conditional input to a 2D
diffusion model. Our approach innovatively incorporates 3D semantic multi-plane
images (MPIs) to provide comprehensive and spatially aligned 3D scene
descriptions for conditioning. As a result, SyntheOcc can generate
photorealistic multi-view images and videos that faithfully align with the
given geometric labels (semantics in 3D voxel space). Extensive qualitative and
quantitative evaluations of SyntheOcc on the nuScenes dataset prove its
effectiveness in generating controllable occupancy datasets that serve as an
effective data augmentation to perception models.
66fd0dad7fbd5d307648b010
Leheng Li
claimed_verified
2024-10-02T12:44:45.347Z
6343e37f73b4f9cedab1c846
/avatars/2638af4626e8a4e3a95f845b94ad94f6.svg
Leheng Li
user
lilelife
66fd0dad7fbd5d307648b011
Weichao Qiu
66fd0dad7fbd5d307648b012
Yingjie Cai
66fd0dad7fbd5d307648b013
Xu Yan
66fd0dad7fbd5d307648b014
Qing Lian
66fd0dad7fbd5d307648b015
Bingbing Liu
66fd0dad7fbd5d307648b016
Ying-Cong Chen
SyntheOcc: Synthesize Geometric-Controlled Street View Images through 3D
  Semantic MPIs
The advancement of autonomous driving is increasingly reliant on high-quality
annotated datasets, especially in the task of 3D occupancy prediction, where
the occupancy labels require dense 3D annotation with significant human effort.
In this paper, we propose SyntheOcc, which denotes a diffusion model that
Synthesize photorealistic and geometric-controlled images by conditioning
Occupancy labels in driving scenarios. This yields an unlimited amount of
diverse, annotated, and controllable datasets for applications like training
perception models and simulation. SyntheOcc addresses the critical challenge of
how to efficiently encode 3D geometric information as conditional input to a 2D
diffusion model. Our approach innovatively incorporates 3D semantic multi-plane
images (MPIs) to provide comprehensive and spatially aligned 3D scene
descriptions for conditioning. As a result, SyntheOcc can generate
photorealistic multi-view images and videos that faithfully align with the
given geometric labels (semantics in 3D voxel space). Extensive qualitative and
quantitative evaluations of SyntheOcc on the nuScenes dataset prove its
effectiveness in generating controllable occupancy datasets that serve as an
effective data augmentation to perception models.
SyntheOcc, a diffusion model using 3D semantic multi-plane images, generates photorealistic and aligned 3D occupancy datasets for autonomous driving perception models.
diffusion model 3D occupancy prediction photorealistic images geometric-controlled images conditional input 3D semantic multi-plane images 3D voxel space controllable datasets data augmentation nuScenes dataset