X-Streamer: Unified Human World Modeling with Audiovisual Interaction
We introduce X-Streamer, an end-to-end multimodal human world modeling
framework for building digital human agents capable of infinite interactions
across text, speech, and video within a single unified architecture. Starting
from a single portrait, X-Streamer enables real-time, open-ended video calls
driven by streaming multimodal inputs. At its core is a Thinker-Actor
dual-transformer architecture that unifies multimodal understanding and
generation, turning a static portrait into persistent and intelligent
audiovisual interactions. The Thinker module perceives and reasons over
streaming user inputs, while its hidden states are translated by the Actor into
synchronized multimodal streams in real time. Concretely, the Thinker leverages
a pretrained large language-speech model, while the Actor employs a chunk-wise
autoregressive diffusion model that cross-attends to the Thinker's hidden
states to produce time-aligned multimodal responses with interleaved discrete
text and audio tokens and continuous video latents. To ensure long-horizon
stability, we design inter- and intra-chunk attentions with time-aligned
multimodal positional embeddings for fine-grained cross-modality alignment and
context retention, further reinforced by chunk-wise diffusion forcing and
global identity referencing. X-Streamer runs in real time on two A100 GPUs,
sustaining hours-long consistent video chat experiences from arbitrary
portraits and paving the way toward unified world modeling of interactive
digital humans.
68d9ed0d0177a6054b013a9f
You Xie
68d9ed0d0177a6054b013aa0
Tianpei Gu
claimed_verified
2025-09-29T05:58:34.638Z
6435ec65f3b08e267d9c9bbd
/avatars/7e846e0c91241898171aa35efd7264c0.svg
Tianpei Gu
user
gutianpei
68d9ed0d0177a6054b013aa1
Zenan Li
68d9ed0d0177a6054b013aa2
Chenxu Zhang
68d9ed0d0177a6054b013aa3
Guoxian Song
68d9ed0d0177a6054b013aa4
Xiaochen Zhao
68d9ed0d0177a6054b013aa5
Chao Liang
68d9ed0d0177a6054b013aa6
Jianwen Jiang
68d9ed0d0177a6054b013aa7
Hongyi Xu
68d9ed0d0177a6054b013aa8
Linjie Luo
X-Streamer: Unified Human World Modeling with Audiovisual Interaction
We introduce X-Streamer, an end-to-end multimodal human world modeling
framework for building digital human agents capable of infinite interactions
across text, speech, and video within a single unified architecture. Starting
from a single portrait, X-Streamer enables real-time, open-ended video calls
driven by streaming multimodal inputs. At its core is a Thinker-Actor
dual-transformer architecture that unifies multimodal understanding and
generation, turning a static portrait into persistent and intelligent
audiovisual interactions. The Thinker module perceives and reasons over
streaming user inputs, while its hidden states are translated by the Actor into
synchronized multimodal streams in real time. Concretely, the Thinker leverages
a pretrained large language-speech model, while the Actor employs a chunk-wise
autoregressive diffusion model that cross-attends to the Thinker's hidden
states to produce time-aligned multimodal responses with interleaved discrete
text and audio tokens and continuous video latents. To ensure long-horizon
stability, we design inter- and intra-chunk attentions with time-aligned
multimodal positional embeddings for fine-grained cross-modality alignment and
context retention, further reinforced by chunk-wise diffusion forcing and
global identity referencing. X-Streamer runs in real time on two A100 GPUs,
sustaining hours-long consistent video chat experiences from arbitrary
portraits and paving the way toward unified world modeling of interactive
digital humans.
X-Streamer is a unified multimodal framework using dual-transformer architecture for real-time, open-ended interactions across text, speech, and video, leveraging large language-speech models and autoregressive diffusion models.
multimodal human world modeling Thinker-Actor dual-transformer architecture Thinker module Actor module pretrained large language-speech model chunk-wise autoregressive diffusion model cross-attention time-aligned multimodal responses discrete text and audio tokens continuous video latents inter-chunk attentions intra-chunk attentions time-aligned multimodal positional embeddings chunk-wise diffusion forcing global identity referencing