StreamDiT: Real-Time Streaming Text-to-Video Generation
Recently, great progress has been achieved in text-to-video (T2V) generation
by scaling transformer-based diffusion models to billions of parameters, which
can generate high-quality videos. However, existing models typically produce
only short clips offline, restricting their use cases in interactive and
real-time applications. This paper addresses these challenges by proposing
StreamDiT, a streaming video generation model. StreamDiT training is based on
flow matching by adding a moving buffer. We design mixed training with
different partitioning schemes of buffered frames to boost both content
consistency and visual quality. StreamDiT modeling is based on adaLN DiT with
varying time embedding and window attention. To practice the proposed method,
we train a StreamDiT model with 4B parameters. In addition, we propose a
multistep distillation method tailored for StreamDiT. Sampling distillation is
performed in each segment of a chosen partitioning scheme. After distillation,
the total number of function evaluations (NFEs) is reduced to the number of
chunks in a buffer. Finally, our distilled model reaches real-time performance
at 16 FPS on one GPU, which can generate video streams at 512p resolution. We
evaluate our method through both quantitative metrics and human evaluation. Our
model enables real-time applications, e.g. streaming generation, interactive
generation, and video-to-video. We provide video results and more examples in
our project website: <a href="https://cumulo-autumn.github.io/StreamDiT/">this
https URL.</a>
686ca04e364e2ad167eb543c
Akio Kodaira
claimed_verified
2025-07-08T08:06:07.401Z
63fedca388b9695964c33ad8
https://cdn-avatars.huggingface.co/v1/production/uploads/63fedca388b9695964c33ad8/XFE8Tm9yRRbo6XdDw7c2O.png
Aki
user
AkiCumulo
686ca04e364e2ad167eb543d
Tingbo Hou
686ca04e364e2ad167eb543e
Ji Hou
686ca04e364e2ad167eb543f
Masayoshi Tomizuka
686ca04e364e2ad167eb5440
Yue Zhao
claimed_verified
2025-07-10T07:10:06.451Z
641cc612f0b71a9743668604
/avatars/ab7d91df8de266fadb4e5632a1f0a37a.svg
Yue (R) Zhao
user
yzhao30
StreamDiT: Real-Time Streaming Text-to-Video Generation
Recently, great progress has been achieved in text-to-video (T2V) generation
by scaling transformer-based diffusion models to billions of parameters, which
can generate high-quality videos. However, existing models typically produce
only short clips offline, restricting their use cases in interactive and
real-time applications. This paper addresses these challenges by proposing
StreamDiT, a streaming video generation model. StreamDiT training is based on
flow matching by adding a moving buffer. We design mixed training with
different partitioning schemes of buffered frames to boost both content
consistency and visual quality. StreamDiT modeling is based on adaLN DiT with
varying time embedding and window attention. To practice the proposed method,
we train a StreamDiT model with 4B parameters. In addition, we propose a
multistep distillation method tailored for StreamDiT. Sampling distillation is
performed in each segment of a chosen partitioning scheme. After distillation,
the total number of function evaluations (NFEs) is reduced to the number of
chunks in a buffer. Finally, our distilled model reaches real-time performance
at 16 FPS on one GPU, which can generate video streams at 512p resolution. We
evaluate our method through both quantitative metrics and human evaluation. Our
model enables real-time applications, e.g. streaming generation, interactive
generation, and video-to-video. We provide video results and more examples in
our project website: <a href="https://cumulo-autumn.github.io/StreamDiT/">this
https URL.</a>
StreamDiT, a streaming video generation model using transformer-based diffusion with flow matching and adaLN DiT, achieves real-time performance at 16 FPS with 4B parameters and multistep distillation.
transformer-based diffusion models flow matching moving buffer mixed training partitioning schemes adaLN DiT time embedding window attention multistep distillation sampling distillation function evaluations real-time performance streaming generation interactive generation video-to-video