TalkingMachines: Real-Time Audio-Driven FaceTime-Style Video via
  Autoregressive Diffusion Models
In this paper, we present TalkingMachines -- an efficient framework that
transforms pretrained video generation models into real-time, audio-driven
character animators. TalkingMachines enables natural conversational experiences
by integrating an audio large language model (LLM) with our video generation
foundation model. Our primary contributions include: (1) We adapt a pretrained
SOTA image-to-video DiT into an audio-driven avatar generation model of 18
billion parameters; (2) We enable infinite video streaming without error
accumulation through asymmetric knowledge distillation from a bidirectional
teacher model into a sparse causal, autoregressive student model; (3) We design
a high-throughput, low-latency inference pipeline incorporating several key
engineering optimizations such as: (a) disaggregation of the DiT and VAE
decoder across separate devices, (b) efficient overlap of inter-device
communication and computation using CUDA streams, (c) elimination of redundant
recomputations to maximize frame-generation throughput. Please see demo videos
here - https://aaxwaz.github.io/TalkingMachines/
684107c6142b5c0b4226025f
Chetwin Low
684107c6142b5c0b42260260
Weimin Wang
TalkingMachines: Real-Time Audio-Driven FaceTime-Style Video via
  Autoregressive Diffusion Models
In this paper, we present TalkingMachines -- an efficient framework that
transforms pretrained video generation models into real-time, audio-driven
character animators. TalkingMachines enables natural conversational experiences
by integrating an audio large language model (LLM) with our video generation
foundation model. Our primary contributions include: (1) We adapt a pretrained
SOTA image-to-video DiT into an audio-driven avatar generation model of 18
billion parameters; (2) We enable infinite video streaming without error
accumulation through asymmetric knowledge distillation from a bidirectional
teacher model into a sparse causal, autoregressive student model; (3) We design
a high-throughput, low-latency inference pipeline incorporating several key
engineering optimizations such as: (a) disaggregation of the DiT and VAE
decoder across separate devices, (b) efficient overlap of inter-device
communication and computation using CUDA streams, (c) elimination of redundant
recomputations to maximize frame-generation throughput. Please see demo videos
here - https://aaxwaz.github.io/TalkingMachines/
TalkingMachines transforms a pretrained image-to-video model into an audio-driven avatar generator, supports infinite video streaming, and uses engineering optimizations for real-time performance.
DiT audio large language model asymmetric knowledge distillation bidirectional teacher model sparse causal autoregressive student model inference pipeline CUDA streams frame-generation throughput