ZeroNVS: Zero-Shot 360-Degree View Synthesis from a Single Real Image
We introduce a 3D-aware diffusion model, ZeroNVS, for single-image novel view
synthesis for in-the-wild scenes. While existing methods are designed for
single objects with masked backgrounds, we propose new techniques to address
challenges introduced by in-the-wild multi-object scenes with complex
backgrounds. Specifically, we train a generative prior on a mixture of data
sources that capture object-centric, indoor, and outdoor scenes. To address
issues from data mixture such as depth-scale ambiguity, we propose a novel
camera conditioning parameterization and normalization scheme. Further, we
observe that Score Distillation Sampling (SDS) tends to truncate the
distribution of complex backgrounds during distillation of 360-degree scenes,
and propose "SDS anchoring" to improve the diversity of synthesized novel
views. Our model sets a new state-of-the-art result in LPIPS on the DTU dataset
in the zero-shot setting, even outperforming methods specifically trained on
DTU. We further adapt the challenging Mip-NeRF 360 dataset as a new benchmark
for single-image novel view synthesis, and demonstrate strong performance in
this setting. Our code and data are at http://kylesargent.github.io/zeronvs/
653f2185b424289c5f839a0d
Kyle Sargent
admin_assigned
2023-10-30T13:37:25.459Z
6451a5d7938967fd069fd135
/avatars/56a00b2e6211f678ec040ea379e7387a.svg
Kyle Sargent
user
ksahug
653f2185b424289c5f839a0e
Zizhang Li
653f2185b424289c5f839a0f
Tanmay Shah
653f2185b424289c5f839a10
Charles Herrmann
admin_assigned
2023-10-30T13:38:03.772Z
64c7f088c864d962eddeaf8c
/avatars/e6364c56e6cb6907b6dd2d34eed83720.svg
Charles Herrmann
user
irwinherrmann
653f2185b424289c5f839a11
Hong-Xing Yu
653f2185b424289c5f839a12
Yunzhi Zhang
admin_assigned
2023-10-30T13:38:31.287Z
63793aee2cb50dda39d6e73b
/avatars/1d7eb1a19ce2e18802e9481200eb03f7.svg
Yunzhi Zhang
user
Yunzhi
653f2185b424289c5f839a13
Eric Ryan Chan
admin_assigned
2023-10-30T13:38:41.553Z
6338dd0db7ce9192552cdf7f
/avatars/d0080d797490926c53690c1b7602d232.svg
Eric Chan
user
ericryanchan
653f2185b424289c5f839a14
Dmitry Lagun
653f2185b424289c5f839a15
Li Fei-Fei
653f2185b424289c5f839a16
Deqing Sun
653f2185b424289c5f839a17
Jiajun Wu
claimed_verified
2023-12-21T09:32:01.925Z
647b7b8345616c3de3627863
/avatars/8196364bffed71086ba9a0a4a62027d3.svg
Jiajun Wu
user
jiajunwu
ZeroNVS: Zero-Shot 360-Degree View Synthesis from a Single Real Image
We introduce a 3D-aware diffusion model, ZeroNVS, for single-image novel view
synthesis for in-the-wild scenes. While existing methods are designed for
single objects with masked backgrounds, we propose new techniques to address
challenges introduced by in-the-wild multi-object scenes with complex
backgrounds. Specifically, we train a generative prior on a mixture of data
sources that capture object-centric, indoor, and outdoor scenes. To address
issues from data mixture such as depth-scale ambiguity, we propose a novel
camera conditioning parameterization and normalization scheme. Further, we
observe that Score Distillation Sampling (SDS) tends to truncate the
distribution of complex backgrounds during distillation of 360-degree scenes,
and propose "SDS anchoring" to improve the diversity of synthesized novel
views. Our model sets a new state-of-the-art result in LPIPS on the DTU dataset
in the zero-shot setting, even outperforming methods specifically trained on
DTU. We further adapt the challenging Mip-NeRF 360 dataset as a new benchmark
for single-image novel view synthesis, and demonstrate strong performance in
this setting. Our code and data are at http://kylesargent.github.io/zeronvs/
A 3D-aware diffusion model, ZeroNVS, addresses challenges in single-image novel view synthesis for complex, in-the-wild scenes, setting new benchmarks and improving diversity in synthesized views.
3D-aware diffusion model novel view synthesis data mixture depth-scale ambiguity camera conditioning normalization scheme Score Distillation Sampling SDS anchoring LPIPS DTU dataset Mip-NeRF 360 dataset