Matrix-Game 2.0: An Open-Source, Real-Time, and Streaming Interactive
  World Model
Recent advances in interactive video generations have demonstrated diffusion
model's potential as world models by capturing complex physical dynamics and
interactive behaviors. However, existing interactive world models depend on
bidirectional attention and lengthy inference steps, severely limiting
real-time performance. Consequently, they are hard to simulate real-world
dynamics, where outcomes must update instantaneously based on historical
context and current actions. To address this, we present Matrix-Game 2.0, an
interactive world model generates long videos on-the-fly via few-step
auto-regressive diffusion. Our framework consists of three key components: (1)
A scalable data production pipeline for Unreal Engine and GTA5 environments to
effectively produce massive amounts (about 1200 hours) of video data with
diverse interaction annotations; (2) An action injection module that enables
frame-level mouse and keyboard inputs as interactive conditions; (3) A few-step
distillation based on the casual architecture for real-time and streaming video
generation. Matrix Game 2.0 can generate high-quality minute-level videos
across diverse scenes at an ultra-fast speed of 25 FPS. We open-source our
model weights and codebase to advance research in interactive world modeling.
68a3e934b65388761d0744e8
Xianglong He
68a3e934b65388761d0744e9
Chunli Peng
68a3e934b65388761d0744ea
Zexiang Liu
68a3e934b65388761d0744eb
Boyang Wang
68a3e934b65388761d0744ec
Yifan Zhang
68a3e934b65388761d0744ed
Qi Cui
68a3e934b65388761d0744ee
Fei Kang
68a3e934b65388761d0744ef
Biao Jiang
68a3e934b65388761d0744f0
Mengyin An
68a3e934b65388761d0744f1
Yangyang Ren
68a3e934b65388761d0744f2
Baixin Xu
68a3e934b65388761d0744f3
Hao-Xiang Guo
68a3e934b65388761d0744f4
Kaixiong Gong
68a3e934b65388761d0744f5
Cyrus Wu
68a3e934b65388761d0744f6
Wei Li
68a3e934b65388761d0744f7
Xuchen Song
68a3e934b65388761d0744f8
Yang Liu
68a3e934b65388761d0744f9
Eric Li
68a3e934b65388761d0744fa
Yahui Zhou
Matrix-Game 2.0: An Open-Source, Real-Time, and Streaming Interactive
  World Model
Recent advances in interactive video generations have demonstrated diffusion
model's potential as world models by capturing complex physical dynamics and
interactive behaviors. However, existing interactive world models depend on
bidirectional attention and lengthy inference steps, severely limiting
real-time performance. Consequently, they are hard to simulate real-world
dynamics, where outcomes must update instantaneously based on historical
context and current actions. To address this, we present Matrix-Game 2.0, an
interactive world model generates long videos on-the-fly via few-step
auto-regressive diffusion. Our framework consists of three key components: (1)
A scalable data production pipeline for Unreal Engine and GTA5 environments to
effectively produce massive amounts (about 1200 hours) of video data with
diverse interaction annotations; (2) An action injection module that enables
frame-level mouse and keyboard inputs as interactive conditions; (3) A few-step
distillation based on the casual architecture for real-time and streaming video
generation. Matrix Game 2.0 can generate high-quality minute-level videos
across diverse scenes at an ultra-fast speed of 25 FPS. We open-source our
model weights and codebase to advance research in interactive world modeling.
Matrix-Game 2.0 generates real-time interactive videos using few-step auto-regressive diffusion, addressing the limitations of lengthy inference in existing models.
diffusion model world models bidirectional attention interactive world models few-step auto-regressive diffusion scalable data production pipeline Unreal Engine GTA5 action injection module frame-level mouse and keyboard inputs few-step distillation casual architecture real-time video generation streaming video generation