InterActHuman: Multi-Concept Human Animation with Layout-Aligned Audio
  Conditions
End-to-end human animation with rich multi-modal conditions, e.g., text,
image and audio has achieved remarkable advancements in recent years. However,
most existing methods could only animate a single subject and inject conditions
in a global manner, ignoring scenarios that multiple concepts could appears in
the same video with rich human-human interactions and human-object
interactions. Such global assumption prevents precise and per-identity control
of multiple concepts including humans and objects, therefore hinders
applications. In this work, we discard the single-entity assumption and
introduce a novel framework that enforces strong, region-specific binding of
conditions from modalities to each identity's spatiotemporal footprint. Given
reference images of multiple concepts, our method could automatically infer
layout information by leveraging a mask predictor to match appearance cues
between the denoised video and each reference appearance. Furthermore, we
inject local audio condition into its corresponding region to ensure
layout-aligned modality matching in a iterative manner. This design enables the
high-quality generation of controllable multi-concept human-centric videos.
Empirical results and ablation studies validate the effectiveness of our
explicit layout control for multi-modal conditions compared to implicit
counterparts and other existing methods.
684a49fa9b38e1e5a33a6884
Zhenzhi Wang
claimed_verified
2025-06-15T07:04:02.011Z
6519346a186bc3b6997c1aaf
/avatars/8981bd278962da50f9bbfb92c2abe2bf.svg
Zhenzhi Wang
user
zhenzhiwang
684a49fa9b38e1e5a33a6885
Jiaqi Yang
684a49fa9b38e1e5a33a6886
Jianwen Jiang
684a49fa9b38e1e5a33a6887
Chao Liang
claimed_verified
2025-06-17T13:07:07.393Z
66d7ca4c0429a62c38c46bf5
https://cdn-avatars.huggingface.co/v1/production/uploads/66d7ca4c0429a62c38c46bf5/N3puSDlwFA8WOxjHjObrj.jpeg
liangdebugger
user
chao0412
684a49fa9b38e1e5a33a6888
Gaojie Lin
claimed_verified
2025-06-12T12:41:52.916Z
64802fcdcc9e514b3b031244
/avatars/cc5979008bdb21a2be9575865dce909b.svg
Gaojie Lin
user
lingaojie
684a49fa9b38e1e5a33a6889
Zerong Zheng
684a49fa9b38e1e5a33a688a
Ceyuan Yang
684a49fa9b38e1e5a33a688b
Dahua Lin
InterActHuman: Multi-Concept Human Animation with Layout-Aligned Audio
  Conditions
End-to-end human animation with rich multi-modal conditions, e.g., text,
image and audio has achieved remarkable advancements in recent years. However,
most existing methods could only animate a single subject and inject conditions
in a global manner, ignoring scenarios that multiple concepts could appears in
the same video with rich human-human interactions and human-object
interactions. Such global assumption prevents precise and per-identity control
of multiple concepts including humans and objects, therefore hinders
applications. In this work, we discard the single-entity assumption and
introduce a novel framework that enforces strong, region-specific binding of
conditions from modalities to each identity's spatiotemporal footprint. Given
reference images of multiple concepts, our method could automatically infer
layout information by leveraging a mask predictor to match appearance cues
between the denoised video and each reference appearance. Furthermore, we
inject local audio condition into its corresponding region to ensure
layout-aligned modality matching in a iterative manner. This design enables the
high-quality generation of controllable multi-concept human-centric videos.
Empirical results and ablation studies validate the effectiveness of our
explicit layout control for multi-modal conditions compared to implicit
counterparts and other existing methods.
A new framework enables precise, per-identity control of multiple concepts in end-to-end human animation by enforcing region-specific binding of multi-modal conditions.
human animation multi-modal conditions mask predictor denoised video layout information local audio condition controllable multi-concept videos explicit layout control