MIDAS: Multimodal Interactive Digital-human Synthesis via Real-time
  Autoregressive Video Generation
Recently, interactive digital human video generation has attracted widespread
attention and achieved remarkable progress. However, building such a practical
system that can interact with diverse input signals in real time remains
challenging to existing methods, which often struggle with high latency, heavy
computational cost, and limited controllability. In this work, we introduce an
autoregressive video generation framework that enables interactive multimodal
control and low-latency extrapolation in a streaming manner. With minimal
modifications to a standard large language model (LLM), our framework accepts
multimodal condition encodings including audio, pose, and text, and outputs
spatially and semantically coherent representations to guide the denoising
process of a diffusion head. To support this, we construct a large-scale
dialogue dataset of approximately 20,000 hours from multiple sources, providing
rich conversational scenarios for training. We further introduce a deep
compression autoencoder with up to 64times reduction ratio, which
effectively alleviates the long-horizon inference burden of the autoregressive
model. Extensive experiments on duplex conversation, multilingual human
synthesis, and interactive world model highlight the advantages of our approach
in low latency, high efficiency, and fine-grained multimodal controllability.
68b008c3245176306494cd22
Ming Chen
claimed_verified
2025-08-28T08:49:49.732Z
6507fbecffc738079ca592bf
/avatars/1cb0f39ac6dc2dba2292846a8d7746da.svg
Ming Chen
user
ChenMing-thu14
68b008c3245176306494cd23
Liyuan Cui
admin_assigned
2025-08-28T18:56:54.892Z
67b2a085a8f5fdc2faf61c6c
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/9ixmIHL3dqGpmj-ffYv7S.png
liyuan cui
user
Cuiliyuan
68b008c3245176306494cd24
Wenyuan Zhang
claimed_verified
2025-09-12T16:21:52.049Z
64d0528459503263d9fb2a2d
/avatars/902b44ccbc5074fcf1fa7da373b38f9f.svg
Zhang Wenyuan
user
zParquet
68b008c3245176306494cd25
Haoxian Zhang
68b008c3245176306494cd26
Yan Zhou
68b008c3245176306494cd27
Xiaohan Li
68b008c3245176306494cd28
Xiaoqiang Liu
68b008c3245176306494cd29
Pengfei Wan
MIDAS: Multimodal Interactive Digital-human Synthesis via Real-time
  Autoregressive Video Generation
Recently, interactive digital human video generation has attracted widespread
attention and achieved remarkable progress. However, building such a practical
system that can interact with diverse input signals in real time remains
challenging to existing methods, which often struggle with high latency, heavy
computational cost, and limited controllability. In this work, we introduce an
autoregressive video generation framework that enables interactive multimodal
control and low-latency extrapolation in a streaming manner. With minimal
modifications to a standard large language model (LLM), our framework accepts
multimodal condition encodings including audio, pose, and text, and outputs
spatially and semantically coherent representations to guide the denoising
process of a diffusion head. To support this, we construct a large-scale
dialogue dataset of approximately 20,000 hours from multiple sources, providing
rich conversational scenarios for training. We further introduce a deep
compression autoencoder with up to 64times reduction ratio, which
effectively alleviates the long-horizon inference burden of the autoregressive
model. Extensive experiments on duplex conversation, multilingual human
synthesis, and interactive world model highlight the advantages of our approach
in low latency, high efficiency, and fine-grained multimodal controllability.
An autoregressive video generation framework with multimodal control and low-latency extrapolation uses a modified large language model and a deep compression autoencoder to achieve high efficiency and fine-grained controllability.
autoregressive video generation multimodal control low-latency extrapolation large language model multimodal condition encodings denoising process diffusion head deep compression autoencoder duplex conversation multilingual human synthesis interactive world model