Viewpoint Textual Inversion: Unleashing Novel View Synthesis with
  Pretrained 2D Diffusion Models
Text-to-image diffusion models understand spatial relationship between
objects, but do they represent the true 3D structure of the world from only 2D
supervision? We demonstrate that yes, 3D knowledge is encoded in 2D image
diffusion models like Stable Diffusion, and we show that this structure can be
exploited for 3D vision tasks. Our method, Viewpoint Neural Textual Inversion
(ViewNeTI), controls the 3D viewpoint of objects in generated images from
frozen diffusion models. We train a small neural mapper to take camera
viewpoint parameters and predict text encoder latents; the latents then
condition the diffusion generation process to produce images with the desired
camera viewpoint.
  ViewNeTI naturally addresses Novel View Synthesis (NVS). By leveraging the
frozen diffusion model as a prior, we can solve NVS with very few input views;
we can even do single-view novel view synthesis. Our single-view NVS
predictions have good semantic details and photorealism compared to prior
methods. Our approach is well suited for modeling the uncertainty inherent in
sparse 3D vision problems because it can efficiently generate diverse samples.
Our view-control mechanism is general, and can even change the camera view in
images generated by user-defined prompts.
6507e8ff04d04d653d088f05
James Burgess
claimed_verified
2023-09-18T17:14:33.118Z
650871aeb44445e9b3625c7b
https://cdn-avatars.huggingface.co/v1/production/uploads/650871aeb44445e9b3625c7b/mtx3EnkuNF4z29IosnhaQ.png
James Burgess
user
jmhb
6507e8ff04d04d653d088f06
Kuan-Chieh Wang
admin_assigned
2023-09-18T11:04:08.815Z
648ca58a39d2584ee47efef6
https://cdn-avatars.huggingface.co/v1/production/uploads/648ca58a39d2584ee47efef6/R7B72bnwc59mdK45rmzYS.png
Kuan-Chieh Wang
user
wangkua1
6507e8ff04d04d653d088f07
Serena Yeung
Viewpoint Textual Inversion: Unleashing Novel View Synthesis with
  Pretrained 2D Diffusion Models
Text-to-image diffusion models understand spatial relationship between
objects, but do they represent the true 3D structure of the world from only 2D
supervision? We demonstrate that yes, 3D knowledge is encoded in 2D image
diffusion models like Stable Diffusion, and we show that this structure can be
exploited for 3D vision tasks. Our method, Viewpoint Neural Textual Inversion
(ViewNeTI), controls the 3D viewpoint of objects in generated images from
frozen diffusion models. We train a small neural mapper to take camera
viewpoint parameters and predict text encoder latents; the latents then
condition the diffusion generation process to produce images with the desired
camera viewpoint.
  ViewNeTI naturally addresses Novel View Synthesis (NVS). By leveraging the
frozen diffusion model as a prior, we can solve NVS with very few input views;
we can even do single-view novel view synthesis. Our single-view NVS
predictions have good semantic details and photorealism compared to prior
methods. Our approach is well suited for modeling the uncertainty inherent in
sparse 3D vision problems because it can efficiently generate diverse samples.
Our view-control mechanism is general, and can even change the camera view in
images generated by user-defined prompts.
Viewpoint Neural Textual Inversion (ViewNeTI) leverages 3D information encoded in text-to-image diffusion models to control camera viewpoints and perform novel view synthesis with high photorealism and semantic detail.
diffusion models text-to-image 3D knowledge viewpoint control Viewpoint Neural Textual Inversion ViewNeTI text encoder latents novel view synthesis NVS single-view NVS semantic details photorealism sparse 3D vision diversity generation camera view control