VLOGGER: Multimodal Diffusion for Embodied Avatar Synthesis
We propose VLOGGER, a method for audio-driven human video generation from a
single input image of a person, which builds on the success of recent
generative diffusion models. Our method consists of 1) a stochastic
human-to-3d-motion diffusion model, and 2) a novel diffusion-based architecture
that augments text-to-image models with both spatial and temporal controls.
This supports the generation of high quality video of variable length, easily
controllable through high-level representations of human faces and bodies. In
contrast to previous work, our method does not require training for each
person, does not rely on face detection and cropping, generates the complete
image (not just the face or the lips), and considers a broad spectrum of
scenarios (e.g. visible torso or diverse subject identities) that are critical
to correctly synthesize humans who communicate. We also curate MENTOR, a new
and diverse dataset with 3d pose and expression annotations, one order of
magnitude larger than previous ones (800,000 identities) and with dynamic
gestures, on which we train and ablate our main technical contributions.
  VLOGGER outperforms state-of-the-art methods in three public benchmarks,
considering image quality, identity preservation and temporal consistency while
also generating upper-body gestures. We analyze the performance of VLOGGER with
respect to multiple diversity metrics, showing that our architectural choices
and the use of MENTOR benefit training a fair and unbiased model at scale.
Finally we show applications in video editing and personalization.
65f26b48b06ca1ff489e81fd
Enric Corona
admin_assigned
2024-03-14T08:05:47.063Z
64a4fe7a80d9ac0155ae911a
/avatars/25c96e0ab25eeb183ac278b2f8ff6262.svg
Enric Corona
user
enriccorona
65f26b48b06ca1ff489e81fe
Andrei Zanfir
admin_assigned
2024-03-14T08:05:53.300Z
648cd24dab8e3024f5bbe93b
/avatars/c8bd4b51155cd37fb7e76c97b843d461.svg
Andrei Zanfir
user
Andreiz
65f26b48b06ca1ff489e81ff
Eduard Gabriel Bazavan
65f26b48b06ca1ff489e8200
Nikos Kolotouros
admin_assigned
2024-03-14T08:06:08.779Z
648cd29f8f46d7284544083b
/avatars/53163398f1a629b9838548b808af53f7.svg
Nikos Kolotouros
user
kolotouros
65f26b48b06ca1ff489e8201
Thiemo Alldieck
admin_assigned
2024-03-14T08:06:14.770Z
648c964d39d2584ee47af19c
https://cdn-avatars.huggingface.co/v1/production/uploads/648c964d39d2584ee47af19c/5UEkzDTMY8I3svKjR7kxN.jpeg
Thiemo Alldieck
user
thiemoall
65f26b48b06ca1ff489e8202
Cristian Sminchisescu
VLOGGER: Multimodal Diffusion for Embodied Avatar Synthesis
We propose VLOGGER, a method for audio-driven human video generation from a
single input image of a person, which builds on the success of recent
generative diffusion models. Our method consists of 1) a stochastic
human-to-3d-motion diffusion model, and 2) a novel diffusion-based architecture
that augments text-to-image models with both spatial and temporal controls.
This supports the generation of high quality video of variable length, easily
controllable through high-level representations of human faces and bodies. In
contrast to previous work, our method does not require training for each
person, does not rely on face detection and cropping, generates the complete
image (not just the face or the lips), and considers a broad spectrum of
scenarios (e.g. visible torso or diverse subject identities) that are critical
to correctly synthesize humans who communicate. We also curate MENTOR, a new
and diverse dataset with 3d pose and expression annotations, one order of
magnitude larger than previous ones (800,000 identities) and with dynamic
gestures, on which we train and ablate our main technical contributions.
  VLOGGER outperforms state-of-the-art methods in three public benchmarks,
considering image quality, identity preservation and temporal consistency while
also generating upper-body gestures. We analyze the performance of VLOGGER with
respect to multiple diversity metrics, showing that our architectural choices
and the use of MENTOR benefit training a fair and unbiased model at scale.
Finally we show applications in video editing and personalization.
VLOGGER generates audio-driven human videos from a single image using a diffusion-based method that includes 3D motion and text-to-image models, outperforming existing methods in quality, identity, and consistency.
stochastic human-to-3d-motion diffusion model diffusion-based architecture text-to-image models spatial controls temporal controls VLOGGER MENTOR 3d pose expression annotations image quality identity preservation temporal consistency upper-body gestures diversity metrics fair model unbiased model video editing personalization