Kling-Avatar: Grounding Multimodal Instructions for Cascaded
  Long-Duration Avatar Animation Synthesis
Recent advances in audio-driven avatar video generation have significantly
enhanced audio-visual realism. However, existing methods treat instruction
conditioning merely as low-level tracking driven by acoustic or visual cues,
without modeling the communicative purpose conveyed by the instructions. This
limitation compromises their narrative coherence and character expressiveness.
To bridge this gap, we introduce Kling-Avatar, a novel cascaded framework that
unifies multimodal instruction understanding with photorealistic portrait
generation. Our approach adopts a two-stage pipeline. In the first stage, we
design a multimodal large language model (MLLM) director that produces a
blueprint video conditioned on diverse instruction signals, thereby governing
high-level semantics such as character motion and emotions. In the second
stage, guided by blueprint keyframes, we generate multiple sub-clips in
parallel using a first-last frame strategy. This global-to-local framework
preserves fine-grained details while faithfully encoding the high-level intent
behind multimodal instructions. Our parallel architecture also enables fast and
stable generation of long-duration videos, making it suitable for real-world
applications such as digital human livestreaming and vlogging. To
comprehensively evaluate our method, we construct a benchmark of 375 curated
samples covering diverse instructions and challenging scenarios. Extensive
experiments demonstrate that Kling-Avatar is capable of generating vivid,
fluent, long-duration videos at up to 1080p and 48 fps, achieving superior
performance in lip synchronization accuracy, emotion and dynamic
expressiveness, instruction controllability, identity preservation, and
cross-domain generalization. These results establish Kling-Avatar as a new
benchmark for semantically grounded, high-fidelity audio-driven avatar
synthesis.
68c37f08fc1747b9124039ce
Yikang Ding
68c37f08fc1747b9124039cf
Jiwen Liu
68c37f08fc1747b9124039d0
Wenyuan Zhang
claimed_verified
2025-09-12T16:08:32.137Z
64d0528459503263d9fb2a2d
/avatars/902b44ccbc5074fcf1fa7da373b38f9f.svg
Zhang Wenyuan
user
zParquet
68c37f08fc1747b9124039d1
Zekun Wang
68c37f08fc1747b9124039d2
Wentao Hu
68c37f08fc1747b9124039d3
Liyuan Cui
68c37f08fc1747b9124039d4
Mingming Lao
68c37f08fc1747b9124039d5
Yingchao Shao
68c37f08fc1747b9124039d6
Hui Liu
68c37f08fc1747b9124039d7
Xiaohan Li
68c37f08fc1747b9124039d8
Ming Chen
68c37f08fc1747b9124039d9
Xiaoqiang Liu
68c37f08fc1747b9124039da
Yu-Shen Liu
68c37f08fc1747b9124039db
Pengfei Wan
Kling-Avatar: Grounding Multimodal Instructions for Cascaded
  Long-Duration Avatar Animation Synthesis
Recent advances in audio-driven avatar video generation have significantly
enhanced audio-visual realism. However, existing methods treat instruction
conditioning merely as low-level tracking driven by acoustic or visual cues,
without modeling the communicative purpose conveyed by the instructions. This
limitation compromises their narrative coherence and character expressiveness.
To bridge this gap, we introduce Kling-Avatar, a novel cascaded framework that
unifies multimodal instruction understanding with photorealistic portrait
generation. Our approach adopts a two-stage pipeline. In the first stage, we
design a multimodal large language model (MLLM) director that produces a
blueprint video conditioned on diverse instruction signals, thereby governing
high-level semantics such as character motion and emotions. In the second
stage, guided by blueprint keyframes, we generate multiple sub-clips in
parallel using a first-last frame strategy. This global-to-local framework
preserves fine-grained details while faithfully encoding the high-level intent
behind multimodal instructions. Our parallel architecture also enables fast and
stable generation of long-duration videos, making it suitable for real-world
applications such as digital human livestreaming and vlogging. To
comprehensively evaluate our method, we construct a benchmark of 375 curated
samples covering diverse instructions and challenging scenarios. Extensive
experiments demonstrate that Kling-Avatar is capable of generating vivid,
fluent, long-duration videos at up to 1080p and 48 fps, achieving superior
performance in lip synchronization accuracy, emotion and dynamic
expressiveness, instruction controllability, identity preservation, and
cross-domain generalization. These results establish Kling-Avatar as a new
benchmark for semantically grounded, high-fidelity audio-driven avatar
synthesis.
Kling-Avatar, a cascaded framework, enhances audio-driven avatar video generation by integrating multimodal instruction understanding with photorealistic portrait generation, resulting in high-fidelity, semantically grounded videos.
multimodal large language model blueprint video high-level semantics character motion emotions blueprint keyframes first-last frame strategy global-to-local framework parallel architecture lip synchronization accuracy emotion and dynamic expressiveness instruction controllability identity preservation cross-domain generalization