RealmDreamer: Text-Driven 3D Scene Generation with Inpainting and Depth
  Diffusion
We introduce RealmDreamer, a technique for generation of general
forward-facing 3D scenes from text descriptions. Our technique optimizes a 3D
Gaussian Splatting representation to match complex text prompts. We initialize
these splats by utilizing the state-of-the-art text-to-image generators,
lifting their samples into 3D, and computing the occlusion volume. We then
optimize this representation across multiple views as a 3D inpainting task with
image-conditional diffusion models. To learn correct geometric structure, we
incorporate a depth diffusion model by conditioning on the samples from the
inpainting model, giving rich geometric structure. Finally, we finetune the
model using sharpened samples from image generators. Notably, our technique
does not require video or multi-view data and can synthesize a variety of
high-quality 3D scenes in different styles, consisting of multiple objects. Its
generality additionally allows 3D synthesis from a single image.
66173de961c1ebcda969238b
Jaidev Shriram
admin_assigned
2024-04-18T13:15:53.741Z
6109b9266bb9e045866e3329
/avatars/caade782fd0ecc5a0f3b0bbbc7d16ccb.svg
Jaidev Shriram
user
JaidevShriram
66173de961c1ebcda969238c
Alex Trevithick
admin_assigned
2024-04-18T13:15:59.848Z
656a76839dcedd16d5d4ed49
/avatars/81e4a17f9af8c990dc13bf1a7bb4efbd.svg
Alex Trevithick
user
atrevithick
66173de961c1ebcda969238d
Lingjie Liu
admin_assigned
2024-04-18T13:16:14.360Z
65ef493cd08d92badad38e54
/avatars/734fde502646760eab2c82ea87bfdff0.svg
liulingie
user
liulingjie
66173de961c1ebcda969238e
Ravi Ramamoorthi
RealmDreamer: Text-Driven 3D Scene Generation with Inpainting and Depth
  Diffusion
We introduce RealmDreamer, a technique for generation of general
forward-facing 3D scenes from text descriptions. Our technique optimizes a 3D
Gaussian Splatting representation to match complex text prompts. We initialize
these splats by utilizing the state-of-the-art text-to-image generators,
lifting their samples into 3D, and computing the occlusion volume. We then
optimize this representation across multiple views as a 3D inpainting task with
image-conditional diffusion models. To learn correct geometric structure, we
incorporate a depth diffusion model by conditioning on the samples from the
inpainting model, giving rich geometric structure. Finally, we finetune the
model using sharpened samples from image generators. Notably, our technique
does not require video or multi-view data and can synthesize a variety of
high-quality 3D scenes in different styles, consisting of multiple objects. Its
generality additionally allows 3D synthesis from a single image.
RealmDreamer generates 3D scenes from text descriptions using 3D Gaussian Splatting, image-conditional diffusion models, and depth diffusion models, achieving high-quality and diverse scene synthesis without requiring video or multi-view data.
3D Gaussian Splatting text-to-image generators image-conditional diffusion models depth diffusion models 3D inpainting task sharpened samples