OmniHuman-1.5: Instilling an Active Mind in Avatars via Cognitive
  Simulation
Existing video avatar models can produce fluid human animations, yet they
struggle to move beyond mere physical likeness to capture a character's
authentic essence. Their motions typically synchronize with low-level cues like
audio rhythm, lacking a deeper semantic understanding of emotion, intent, or
context. To bridge this gap, we propose a framework designed to
generate character animations that are not only physically plausible but also
semantically coherent and expressive. Our model, OmniHuman-1.5, is
built upon two key technical contributions. First, we leverage Multimodal Large
Language Models to synthesize a structured textual representation of conditions
that provides high-level semantic guidance. This guidance steers our motion
generator beyond simplistic rhythmic synchronization, enabling the production
of actions that are contextually and emotionally resonant. Second, to ensure
the effective fusion of these multimodal inputs and mitigate inter-modality
conflicts, we introduce a specialized Multimodal DiT architecture with a novel
Pseudo Last Frame design. The synergy of these components allows our model to
accurately interpret the joint semantics of audio, images, and text, thereby
generating motions that are deeply coherent with the character, scene, and
linguistic content. Extensive experiments demonstrate that our model achieves
leading performance across a comprehensive set of metrics, including lip-sync
accuracy, video quality, motion naturalness and semantic consistency with
textual prompts. Furthermore, our approach shows remarkable extensibility to
complex scenarios, such as those involving multi-person and non-human subjects.
Homepage: https://omnihuman-lab.github.io/v1_5/
68ae7568364411bea07df736
Jianwen Jiang
68ae7568364411bea07df737
Weihong Zeng
68ae7568364411bea07df738
Zerong Zheng
68ae7568364411bea07df739
Jiaqi Yang
68ae7568364411bea07df73a
Chao Liang
claimed_verified
2025-08-27T07:10:50.020Z
66d7ca4c0429a62c38c46bf5
https://cdn-avatars.huggingface.co/v1/production/uploads/66d7ca4c0429a62c38c46bf5/N3puSDlwFA8WOxjHjObrj.jpeg
liangdebugger
user
chao0412
68ae7568364411bea07df73b
Wang Liao
68ae7568364411bea07df73c
Han Liang
68ae7568364411bea07df73d
Yuan Zhang
68ae7568364411bea07df73e
Mingyuan Gao
OmniHuman-1.5: Instilling an Active Mind in Avatars via Cognitive
  Simulation
Existing video avatar models can produce fluid human animations, yet they
struggle to move beyond mere physical likeness to capture a character's
authentic essence. Their motions typically synchronize with low-level cues like
audio rhythm, lacking a deeper semantic understanding of emotion, intent, or
context. To bridge this gap, we propose a framework designed to
generate character animations that are not only physically plausible but also
semantically coherent and expressive. Our model, OmniHuman-1.5, is
built upon two key technical contributions. First, we leverage Multimodal Large
Language Models to synthesize a structured textual representation of conditions
that provides high-level semantic guidance. This guidance steers our motion
generator beyond simplistic rhythmic synchronization, enabling the production
of actions that are contextually and emotionally resonant. Second, to ensure
the effective fusion of these multimodal inputs and mitigate inter-modality
conflicts, we introduce a specialized Multimodal DiT architecture with a novel
Pseudo Last Frame design. The synergy of these components allows our model to
accurately interpret the joint semantics of audio, images, and text, thereby
generating motions that are deeply coherent with the character, scene, and
linguistic content. Extensive experiments demonstrate that our model achieves
leading performance across a comprehensive set of metrics, including lip-sync
accuracy, video quality, motion naturalness and semantic consistency with
textual prompts. Furthermore, our approach shows remarkable extensibility to
complex scenarios, such as those involving multi-person and non-human subjects.
Homepage: https://omnihuman-lab.github.io/v1_5/
A framework using Multimodal Large Language Models and a specialized Multimodal DiT architecture generates semantically coherent and expressive character animations from multimodal inputs.
Multimodal Large Language Models Multimodal DiT architecture Pseudo Last Frame design semantic coherence multimodal fusion inter-modality conflicts lip-sync accuracy video quality motion naturalness semantic consistency