OmniHuman-1: Rethinking the Scaling-Up of One-Stage Conditioned Human
  Animation Models
End-to-end human animation, such as audio-driven talking human generation,
has undergone notable advancements in the recent few years. However, existing
methods still struggle to scale up as large general video generation models,
limiting their potential in real applications. In this paper, we propose
OmniHuman, a Diffusion Transformer-based framework that scales up data by
mixing motion-related conditions into the training phase. To this end, we
introduce two training principles for these mixed conditions, along with the
corresponding model architecture and inference strategy. These designs enable
OmniHuman to fully leverage data-driven motion generation, ultimately achieving
highly realistic human video generation. More importantly, OmniHuman supports
various portrait contents (face close-up, portrait, half-body, full-body),
supports both talking and singing, handles human-object interactions and
challenging body poses, and accommodates different image styles. Compared to
existing end-to-end audio-driven methods, OmniHuman not only produces more
realistic videos, but also offers greater flexibility in inputs. It also
supports multiple driving modalities (audio-driven, video-driven and combined
driving signals). Video samples are provided on the ttfamily project page
(https://omnihuman-lab.github.io)
67a1a7a166a8a88726963ef4
Gaojie Lin
admin_assigned
2025-02-04T14:49:22.604Z
64802fcdcc9e514b3b031244
/avatars/cc5979008bdb21a2be9575865dce909b.svg
Gaojie Lin
user
lingaojie
67a1a7a166a8a88726963ef5
Jianwen Jiang
claimed_verified
2025-02-05T10:15:17.797Z
643410509bd5a84b5dca1345
/avatars/597823ce61c7b1a5da77e178820824f6.svg
Jianwen Jiang
user
JianwenJ
67a1a7a166a8a88726963ef6
Jiaqi Yang
claimed_verified
2025-02-14T08:01:37.757Z
6527603743db8c626c467726
/avatars/216df0374b37355d57d2c76e6f08e4d6.svg
yang
user
jiaqi78
67a1a7a166a8a88726963ef7
Zerong Zheng
admin_assigned
2025-02-04T14:50:10.474Z
65b0b6d00648e8d10b609066
/avatars/071d2a99f6d7a4e37d338e58d46c4bc2.svg
ZZerong
user
zerong2
67a1a7a166a8a88726963ef8
Chao Liang
claimed_verified
2025-02-05T10:15:22.815Z
66d7ca4c0429a62c38c46bf5
https://cdn-avatars.huggingface.co/v1/production/uploads/66d7ca4c0429a62c38c46bf5/N3puSDlwFA8WOxjHjObrj.jpeg
liangdebugger
user
chao0412
OmniHuman-1: Rethinking the Scaling-Up of One-Stage Conditioned Human
  Animation Models
End-to-end human animation, such as audio-driven talking human generation,
has undergone notable advancements in the recent few years. However, existing
methods still struggle to scale up as large general video generation models,
limiting their potential in real applications. In this paper, we propose
OmniHuman, a Diffusion Transformer-based framework that scales up data by
mixing motion-related conditions into the training phase. To this end, we
introduce two training principles for these mixed conditions, along with the
corresponding model architecture and inference strategy. These designs enable
OmniHuman to fully leverage data-driven motion generation, ultimately achieving
highly realistic human video generation. More importantly, OmniHuman supports
various portrait contents (face close-up, portrait, half-body, full-body),
supports both talking and singing, handles human-object interactions and
challenging body poses, and accommodates different image styles. Compared to
existing end-to-end audio-driven methods, OmniHuman not only produces more
realistic videos, but also offers greater flexibility in inputs. It also
supports multiple driving modalities (audio-driven, video-driven and combined
driving signals). Video samples are provided on the ttfamily project page
(https://omnihuman-lab.github.io)
OmniHuman, a Diffusion Transformer-based framework, achieves realistic human video generation by integrating motion-related conditions into its training process, supporting various inputs and driving modalities.
Diffusion Transformer motion-related conditions model architecture inference strategy data-driven motion generation human video generation portrait contents human-object interactions body poses image styles audio-driven video-driven combined driving signals