HyperHuman: Hyper-Realistic Human Generation with Latent Structural
  Diffusion
Despite significant advances in large-scale text-to-image models, achieving
hyper-realistic human image generation remains a desirable yet unsolved task.
Existing models like Stable Diffusion and DALL-E 2 tend to generate human
images with incoherent parts or unnatural poses. To tackle these challenges,
our key insight is that human image is inherently structural over multiple
granularities, from the coarse-level body skeleton to fine-grained spatial
geometry. Therefore, capturing such correlations between the explicit
appearance and latent structure in one model is essential to generate coherent
and natural human images. To this end, we propose a unified framework,
HyperHuman, that generates in-the-wild human images of high realism and diverse
layouts. Specifically, 1) we first build a large-scale human-centric dataset,
named HumanVerse, which consists of 340M images with comprehensive annotations
like human pose, depth, and surface normal. 2) Next, we propose a Latent
Structural Diffusion Model that simultaneously denoises the depth and surface
normal along with the synthesized RGB image. Our model enforces the joint
learning of image appearance, spatial relationship, and geometry in a unified
network, where each branch in the model complements to each other with both
structural awareness and textural richness. 3) Finally, to further boost the
visual quality, we propose a Structure-Guided Refiner to compose the predicted
conditions for more detailed generation of higher resolution. Extensive
experiments demonstrate that our framework yields the state-of-the-art
performance, generating hyper-realistic human images under diverse scenarios.
Project Page: https://snap-research.github.io/HyperHuman/
65294d8a0b44378dcc6862a9
Xian Liu
admin_assigned
2023-10-13T15:50:41.663Z
6423f10a3fa01ecba6fca861
/avatars/e4b4c7dddc5cf962ade45d8bdfa10b1a.svg
Xian Liu
user
alvinliu0
65294d8a0b44378dcc6862aa
Jian Ren
extracted_confirmed
2023-10-26T03:40:22.879Z
61f19829233c91cbd2f79e70
/avatars/a0735a94542b4f7cda5aed8bc4be0538.svg
Jian Ren
user
alanspike
65294d8a0b44378dcc6862ab
Aliaksandr Siarohin
admin_assigned
2023-10-13T15:50:49.640Z
64276311eb9a0ed86180715b
/avatars/76f933cd549f10e5e2db379de235d304.svg
Aliaksandr Siarohin
user
aliaksandr-siarohin
65294d8a0b44378dcc6862ac
Ivan Skorokhodov
admin_assigned
2023-10-13T15:50:58.308Z
63610db13c7147fae7de88e3
/avatars/d7e97a16cfee39e1e50d7a5b747876f1.svg
Ivan Skorokhodov
user
universome
65294d8a0b44378dcc6862ad
Yanyu Li
65294d8a0b44378dcc6862ae
Dahua Lin
admin_assigned
2023-10-13T15:51:16.567Z
636317ed80c1a705a6eff396
/avatars/3db090e101b916d9256d0d3e043db71d.svg
Dahua Lin
user
lindahua
65294d8a0b44378dcc6862af
Xihui Liu
claimed_verified
2024-06-03T18:30:36.148Z
65d5ec74cd05bc1eaa125040
/avatars/2de1b1539a86452c2c89570eeb02f5ab.svg
Xihui Liu
user
XihuiLiu
65294d8a0b44378dcc6862b0
Ziwei Liu
admin_assigned
2023-10-13T15:51:36.113Z
62ab1ac1d48b4d8b048a3473
https://cdn-avatars.huggingface.co/v1/production/uploads/1656826685333-62ab1ac1d48b4d8b048a3473.png
Ziwei Liu
user
liuziwei7
65294d8a0b44378dcc6862b1
Sergey Tulyakov
HyperHuman: Hyper-Realistic Human Generation with Latent Structural
  Diffusion
Despite significant advances in large-scale text-to-image models, achieving
hyper-realistic human image generation remains a desirable yet unsolved task.
Existing models like Stable Diffusion and DALL-E 2 tend to generate human
images with incoherent parts or unnatural poses. To tackle these challenges,
our key insight is that human image is inherently structural over multiple
granularities, from the coarse-level body skeleton to fine-grained spatial
geometry. Therefore, capturing such correlations between the explicit
appearance and latent structure in one model is essential to generate coherent
and natural human images. To this end, we propose a unified framework,
HyperHuman, that generates in-the-wild human images of high realism and diverse
layouts. Specifically, 1) we first build a large-scale human-centric dataset,
named HumanVerse, which consists of 340M images with comprehensive annotations
like human pose, depth, and surface normal. 2) Next, we propose a Latent
Structural Diffusion Model that simultaneously denoises the depth and surface
normal along with the synthesized RGB image. Our model enforces the joint
learning of image appearance, spatial relationship, and geometry in a unified
network, where each branch in the model complements to each other with both
structural awareness and textural richness. 3) Finally, to further boost the
visual quality, we propose a Structure-Guided Refiner to compose the predicted
conditions for more detailed generation of higher resolution. Extensive
experiments demonstrate that our framework yields the state-of-the-art
performance, generating hyper-realistic human images under diverse scenarios.
Project Page: https://snap-research.github.io/HyperHuman/
A unified framework, HyperHuman, generates hyper-realistic human images using a Latent Structural Diffusion Model and a Structure-Guided Refiner, leveraging a large-scale human-centric dataset for detailed, high-quality image synthesis.
Latent Structural Diffusion Model Structure-Guided Refiner HumanVerse human-centric dataset human pose depth surface normal joint learning image appearance spatial relationship geometry unified network structural awareness textural richness high realism diverse scenarios