Single-Image 3D Human Digitization with Shape-Guided Diffusion
We present an approach to generate a 360-degree view of a person with a
consistent, high-resolution appearance from a single input image. NeRF and its
variants typically require videos or images from different viewpoints. Most
existing approaches taking monocular input either rely on ground-truth 3D scans
for supervision or lack 3D consistency. While recent 3D generative models show
promise of 3D consistent human digitization, these approaches do not generalize
well to diverse clothing appearances, and the results lack photorealism. Unlike
existing work, we utilize high-capacity 2D diffusion models pretrained for
general image synthesis tasks as an appearance prior of clothed humans. To
achieve better 3D consistency while retaining the input identity, we
progressively synthesize multiple views of the human in the input image by
inpainting missing regions with shape-guided diffusion conditioned on
silhouette and surface normal. We then fuse these synthesized multi-view images
via inverse rendering to obtain a fully textured high-resolution 3D mesh of the
given person. Experiments show that our approach outperforms prior methods and
achieves photorealistic 360-degree synthesis of a wide range of clothed humans
with complex textures from a single image.
6555b04e4f361968f0fcebc0
Badour AlBahar
claimed_verified
2023-11-16T15:51:07.035Z
635d39efd6bbbf54119a76a0
/avatars/97e2dca56fe72ed7406acc52180be409.svg
Badour Albahar
user
Badour
6555b04e4f361968f0fcebc1
Shunsuke Saito
admin_assigned
2023-11-16T08:12:40.028Z
630688edd37ce67e0e50e548
/avatars/1681e465d7649f67f94e1b69d236cb1e.svg
Shunsuke Saito
user
psyth
6555b04e4f361968f0fcebc2
Hung-Yu Tseng
6555b04e4f361968f0fcebc3
Changil Kim
6555b04e4f361968f0fcebc4
Johannes Kopf
6555b04e4f361968f0fcebc5
Jia-Bin Huang
admin_assigned
2023-11-16T08:14:16.285Z
641c139b73296f7ee256970c
/avatars/5a2550d95e686640242840ad3bd0e680.svg
Jiabin Huang
user
YellowAddice
Single-Image 3D Human Digitization with Shape-Guided Diffusion
We present an approach to generate a 360-degree view of a person with a
consistent, high-resolution appearance from a single input image. NeRF and its
variants typically require videos or images from different viewpoints. Most
existing approaches taking monocular input either rely on ground-truth 3D scans
for supervision or lack 3D consistency. While recent 3D generative models show
promise of 3D consistent human digitization, these approaches do not generalize
well to diverse clothing appearances, and the results lack photorealism. Unlike
existing work, we utilize high-capacity 2D diffusion models pretrained for
general image synthesis tasks as an appearance prior of clothed humans. To
achieve better 3D consistency while retaining the input identity, we
progressively synthesize multiple views of the human in the input image by
inpainting missing regions with shape-guided diffusion conditioned on
silhouette and surface normal. We then fuse these synthesized multi-view images
via inverse rendering to obtain a fully textured high-resolution 3D mesh of the
given person. Experiments show that our approach outperforms prior methods and
achieves photorealistic 360-degree synthesis of a wide range of clothed humans
with complex textures from a single image.
An approach using 2D diffusion models and inverse rendering generates photorealistic 360-degree views of people from single images.
NeRF 3D generative models 2D diffusion models appearance prior shape-guided diffusion silhouette surface normal inverse rendering 3D mesh