StableAvatar: Infinite-Length Audio-Driven Avatar Video Generation
Current diffusion models for audio-driven avatar video generation struggle to
synthesize long videos with natural audio synchronization and identity
consistency. This paper presents StableAvatar, the first end-to-end video
diffusion transformer that synthesizes infinite-length high-quality videos
without post-processing. Conditioned on a reference image and audio,
StableAvatar integrates tailored training and inference modules to enable
infinite-length video generation. We observe that the main reason preventing
existing models from generating long videos lies in their audio modeling. They
typically rely on third-party off-the-shelf extractors to obtain audio
embeddings, which are then directly injected into the diffusion model via
cross-attention. Since current diffusion backbones lack any audio-related
priors, this approach causes severe latent distribution error accumulation
across video clips, leading the latent distribution of subsequent segments to
drift away from the optimal distribution gradually. To address this,
StableAvatar introduces a novel Time-step-aware Audio Adapter that prevents
error accumulation via time-step-aware modulation. During inference, we propose
a novel Audio Native Guidance Mechanism to further enhance the audio
synchronization by leveraging the diffusion's own evolving joint audio-latent
prediction as a dynamic guidance signal. To enhance the smoothness of the
infinite-length videos, we introduce a Dynamic Weighted Sliding-window Strategy
that fuses latent over time. Experiments on benchmarks show the effectiveness
of StableAvatar both qualitatively and quantitatively.
689ab940fab6fdd2e52ac4ab
Shuyuan Tu
admin_assigned
2025-08-13T19:02:50.161Z
66da6972eae491c64243e8f3
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/SgX1j3QGKMk_YFTnM9Wr_.png
Shuyuan Tu
user
FrancisRing
689ab940fab6fdd2e52ac4ac
Yueming Pan
689ab940fab6fdd2e52ac4ad
Yinming Huang
689ab940fab6fdd2e52ac4ae
Xintong Han
689ab940fab6fdd2e52ac4af
Zhen Xing
689ab940fab6fdd2e52ac4b0
Qi Dai
689ab940fab6fdd2e52ac4b1
Chong Luo
689ab940fab6fdd2e52ac4b2
Zuxuan Wu
689ab940fab6fdd2e52ac4b3
Yu-Gang Jiang
StableAvatar: Infinite-Length Audio-Driven Avatar Video Generation
Current diffusion models for audio-driven avatar video generation struggle to
synthesize long videos with natural audio synchronization and identity
consistency. This paper presents StableAvatar, the first end-to-end video
diffusion transformer that synthesizes infinite-length high-quality videos
without post-processing. Conditioned on a reference image and audio,
StableAvatar integrates tailored training and inference modules to enable
infinite-length video generation. We observe that the main reason preventing
existing models from generating long videos lies in their audio modeling. They
typically rely on third-party off-the-shelf extractors to obtain audio
embeddings, which are then directly injected into the diffusion model via
cross-attention. Since current diffusion backbones lack any audio-related
priors, this approach causes severe latent distribution error accumulation
across video clips, leading the latent distribution of subsequent segments to
drift away from the optimal distribution gradually. To address this,
StableAvatar introduces a novel Time-step-aware Audio Adapter that prevents
error accumulation via time-step-aware modulation. During inference, we propose
a novel Audio Native Guidance Mechanism to further enhance the audio
synchronization by leveraging the diffusion's own evolving joint audio-latent
prediction as a dynamic guidance signal. To enhance the smoothness of the
infinite-length videos, we introduce a Dynamic Weighted Sliding-window Strategy
that fuses latent over time. Experiments on benchmarks show the effectiveness
of StableAvatar both qualitatively and quantitatively.
StableAvatar, an end-to-end video diffusion transformer, synthesizes infinite-length high-quality audio-driven avatar videos with natural synchronization and identity consistency using a Time-step-aware Audio Adapter and Audio Native Guidance Mechanism.
diffusion models audio-driven avatar video generation end-to-end video diffusion transformer infinite-length video generation reference image audio embeddings cross-attention diffusion backbones latent distribution error accumulation Time-step-aware Audio Adapter Audio Native Guidance Mechanism Dynamic Weighted Sliding-window Strategy