Loopy: Taming Audio-Driven Portrait Avatar with Long-Term Motion
  Dependency
With the introduction of diffusion-based video generation techniques,
audio-conditioned human video generation has recently achieved significant
breakthroughs in both the naturalness of motion and the synthesis of portrait
details. Due to the limited control of audio signals in driving human motion,
existing methods often add auxiliary spatial signals to stabilize movements,
which may compromise the naturalness and freedom of motion. In this paper, we
propose an end-to-end audio-only conditioned video diffusion model named Loopy.
Specifically, we designed an inter- and intra-clip temporal module and an
audio-to-latents module, enabling the model to leverage long-term motion
information from the data to learn natural motion patterns and improving
audio-portrait movement correlation. This method removes the need for manually
specified spatial motion templates used in existing methods to constrain motion
during inference. Extensive experiments show that Loopy outperforms recent
audio-driven portrait diffusion models, delivering more lifelike and
high-quality results across various scenarios.
66d9211e9e955e0e5d1d7040
Jianwen Jiang
claimed_verified
2025-02-05T10:16:32.675Z
643410509bd5a84b5dca1345
/avatars/597823ce61c7b1a5da77e178820824f6.svg
Jianwen Jiang
user
JianwenJ
66d9211e9e955e0e5d1d7041
Chao Liang
claimed_verified
2024-09-09T10:58:07.640Z
66d7ca4c0429a62c38c46bf5
https://cdn-avatars.huggingface.co/v1/production/uploads/66d7ca4c0429a62c38c46bf5/N3puSDlwFA8WOxjHjObrj.jpeg
liangdebugger
user
chao0412
66d9211e9e955e0e5d1d7042
Jiaqi Yang
claimed_verified
2024-09-06T07:06:27.786Z
6527603743db8c626c467726
/avatars/216df0374b37355d57d2c76e6f08e4d6.svg
yang
user
jiaqi78
66d9211e9e955e0e5d1d7043
Gaojie Lin
claimed_verified
2025-06-12T12:46:12.348Z
64802fcdcc9e514b3b031244
/avatars/cc5979008bdb21a2be9575865dce909b.svg
Gaojie Lin
user
lingaojie
66d9211e9e955e0e5d1d7044
Tianyun Zhong
admin_assigned
2024-09-05T08:50:36.107Z
6393200e7a88f87c10056c43
/avatars/4ddde5cc4e5780e55d12ce0129234365.svg
zhongtianyun
user
tianyumyum
66d9211e9e955e0e5d1d7045
Yanbo Zheng
Loopy: Taming Audio-Driven Portrait Avatar with Long-Term Motion
  Dependency
With the introduction of diffusion-based video generation techniques,
audio-conditioned human video generation has recently achieved significant
breakthroughs in both the naturalness of motion and the synthesis of portrait
details. Due to the limited control of audio signals in driving human motion,
existing methods often add auxiliary spatial signals to stabilize movements,
which may compromise the naturalness and freedom of motion. In this paper, we
propose an end-to-end audio-only conditioned video diffusion model named Loopy.
Specifically, we designed an inter- and intra-clip temporal module and an
audio-to-latents module, enabling the model to leverage long-term motion
information from the data to learn natural motion patterns and improving
audio-portrait movement correlation. This method removes the need for manually
specified spatial motion templates used in existing methods to constrain motion
during inference. Extensive experiments show that Loopy outperforms recent
audio-driven portrait diffusion models, delivering more lifelike and
high-quality results across various scenarios.
An audio-only conditioned video diffusion model, Loopy, improves natural and high-quality human video generation without auxiliary spatial signals.
diffusion-based video generation audio-conditioned human video generation motion portrait details audio signals inter- and intra-clip temporal module audio-to-latents module long-term motion information natural motion patterns audio-portrait movement correlation video diffusion model spatial motion templates lifelike results