HuMo: Human-Centric Video Generation via Collaborative Multi-Modal
  Conditioning
Human-Centric Video Generation (HCVG) methods seek to synthesize human videos
from multimodal inputs, including text, image, and audio. Existing methods
struggle to effectively coordinate these heterogeneous modalities due to two
challenges: the scarcity of training data with paired triplet conditions and
the difficulty of collaborating the sub-tasks of subject preservation and
audio-visual sync with multimodal inputs. In this work, we present HuMo, a
unified HCVG framework for collaborative multimodal control. For the first
challenge, we construct a high-quality dataset with diverse and paired text,
reference images, and audio. For the second challenge, we propose a two-stage
progressive multimodal training paradigm with task-specific strategies. For the
subject preservation task, to maintain the prompt following and visual
generation abilities of the foundation model, we adopt the minimal-invasive
image injection strategy. For the audio-visual sync task, besides the commonly
adopted audio cross-attention layer, we propose a focus-by-predicting strategy
that implicitly guides the model to associate audio with facial regions. For
joint learning of controllabilities across multimodal inputs, building on
previously acquired capabilities, we progressively incorporate the audio-visual
sync task. During inference, for flexible and fine-grained multimodal control,
we design a time-adaptive Classifier-Free Guidance strategy that dynamically
adjusts guidance weights across denoising steps. Extensive experimental results
demonstrate that HuMo surpasses specialized state-of-the-art methods in
sub-tasks, establishing a unified framework for collaborative
multimodal-conditioned HCVG. Project Page:
https://phantom-video.github.io/HuMo.
68c2624629b8ec9932cd09ea
Liyang Chen
claimed_verified
2025-09-12T16:12:31.849Z
65e9866cdde63935299cead7
https://cdn-avatars.huggingface.co/v1/production/uploads/65e9866cdde63935299cead7/yBF_iDl0ga6lfMp3w-TTz.jpeg
Liyang Chen
user
leoniuschen
68c2624629b8ec9932cd09eb
Tianxiang Ma
68c2624629b8ec9932cd09ec
Jiawei Liu
68c2624629b8ec9932cd09ed
Bingchuan Li
68c2624629b8ec9932cd09ee
Zhuowei Chen
68c2624629b8ec9932cd09ef
Lijie Liu
68c2624629b8ec9932cd09f0
Xu He
68c2624629b8ec9932cd09f1
Gen Li
68c2624629b8ec9932cd09f2
Qian He
68c2624629b8ec9932cd09f3
Zhiyong Wu
HuMo: Human-Centric Video Generation via Collaborative Multi-Modal
  Conditioning
Human-Centric Video Generation (HCVG) methods seek to synthesize human videos
from multimodal inputs, including text, image, and audio. Existing methods
struggle to effectively coordinate these heterogeneous modalities due to two
challenges: the scarcity of training data with paired triplet conditions and
the difficulty of collaborating the sub-tasks of subject preservation and
audio-visual sync with multimodal inputs. In this work, we present HuMo, a
unified HCVG framework for collaborative multimodal control. For the first
challenge, we construct a high-quality dataset with diverse and paired text,
reference images, and audio. For the second challenge, we propose a two-stage
progressive multimodal training paradigm with task-specific strategies. For the
subject preservation task, to maintain the prompt following and visual
generation abilities of the foundation model, we adopt the minimal-invasive
image injection strategy. For the audio-visual sync task, besides the commonly
adopted audio cross-attention layer, we propose a focus-by-predicting strategy
that implicitly guides the model to associate audio with facial regions. For
joint learning of controllabilities across multimodal inputs, building on
previously acquired capabilities, we progressively incorporate the audio-visual
sync task. During inference, for flexible and fine-grained multimodal control,
we design a time-adaptive Classifier-Free Guidance strategy that dynamically
adjusts guidance weights across denoising steps. Extensive experimental results
demonstrate that HuMo surpasses specialized state-of-the-art methods in
sub-tasks, establishing a unified framework for collaborative
multimodal-conditioned HCVG. Project Page:
https://phantom-video.github.io/HuMo.
HuMo is a unified framework for human-centric video generation that addresses challenges in multimodal control through a two-stage training paradigm and novel strategies for subject preservation and audio-visual synchronization.
human-centric video generation multimodal inputs text image audio high-quality dataset two-stage progressive multimodal training minimal-invasive image injection audio cross-attention layer focus-by-predicting strategy time-adaptive Classifier-Free Guidance denoising steps