Synthesizing Moving People with 3D Control
In this paper, we present a diffusion model-based framework for animating
people from a single image for a given target 3D motion sequence. Our approach
has two core components: a) learning priors about invisible parts of the human
body and clothing, and b) rendering novel body poses with proper clothing and
texture. For the first part, we learn an in-filling diffusion model to
hallucinate unseen parts of a person given a single image. We train this model
on texture map space, which makes it more sample-efficient since it is
invariant to pose and viewpoint. Second, we develop a diffusion-based rendering
pipeline, which is controlled by 3D human poses. This produces realistic
renderings of novel poses of the person, including clothing, hair, and
plausible in-filling of unseen regions. This disentangled approach allows our
method to generate a sequence of images that are faithful to the target motion
in the 3D pose and, to the input image in terms of visual similarity. In
addition to that, the 3D control allows various synthetic camera trajectories
to render a person. Our experiments show that our method is resilient in
generating prolonged motions and varied challenging and complex poses compared
to prior methods. Please check our website for more details:
https://boyiliee.github.io/3DHM.github.io/.
65ade45aa8f716b32e027084
Boyi Li
admin_assigned
2024-01-22T13:11:55.391Z
620dd3888528f797e88cb9b5
/avatars/af04728788d78fe7d6375e19e32a535e.svg
Boyi Li
user
Boyiliee
65ade45aa8f716b32e027085
Jathushan Rajasegaran
admin_assigned
2024-01-22T13:12:02.287Z
63895b3a43d8b0797a0d0406
/avatars/04d9952dff70f85d4e481ec80ae818cb.svg
Jathushan Rajasegaran
user
brjathu
65ade45aa8f716b32e027086
Yossi Gandelsman
claimed_verified
2024-01-22T10:28:47.504Z
62f6942205ca68c0e0ff7b97
/avatars/b0600de531b4c88f4aa7c30c5ceb06e1.svg
Yossi Gandelsman
user
yossig
65ade45aa8f716b32e027087
Alexei A. Efros
65ade45aa8f716b32e027088
Jitendra Malik
admin_assigned
2024-01-22T13:12:15.276Z
65369a95605a07338de78ab0
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/sGFjOjLT2akN-sn5beVWL.jpeg
Jitendra Malik 
user
jitendra1995
Synthesizing Moving People with 3D Control
In this paper, we present a diffusion model-based framework for animating
people from a single image for a given target 3D motion sequence. Our approach
has two core components: a) learning priors about invisible parts of the human
body and clothing, and b) rendering novel body poses with proper clothing and
texture. For the first part, we learn an in-filling diffusion model to
hallucinate unseen parts of a person given a single image. We train this model
on texture map space, which makes it more sample-efficient since it is
invariant to pose and viewpoint. Second, we develop a diffusion-based rendering
pipeline, which is controlled by 3D human poses. This produces realistic
renderings of novel poses of the person, including clothing, hair, and
plausible in-filling of unseen regions. This disentangled approach allows our
method to generate a sequence of images that are faithful to the target motion
in the 3D pose and, to the input image in terms of visual similarity. In
addition to that, the 3D control allows various synthetic camera trajectories
to render a person. Our experiments show that our method is resilient in
generating prolonged motions and varied challenging and complex poses compared
to prior methods. Please check our website for more details:
https://boyiliee.github.io/3DHM.github.io/.
A diffusion model-based framework animates people from a single image using learned priors for unseen parts and a diffusion-based rendering pipeline controlled by 3D human poses.
diffusion model in-filling diffusion model texture map space diffusion-based rendering pipeline 3D human poses realistic renderings visual similarity synthetic camera trajectories