Diffusion Models Are Real-Time Game Engines
We present GameNGen, the first game engine powered entirely by a neural model
that enables real-time interaction with a complex environment over long
trajectories at high quality. GameNGen can interactively simulate the classic
game DOOM at over 20 frames per second on a single TPU. Next frame prediction
achieves a PSNR of 29.4, comparable to lossy JPEG compression. Human raters are
only slightly better than random chance at distinguishing short clips of the
game from clips of the simulation. GameNGen is trained in two phases: (1) an
RL-agent learns to play the game and the training sessions are recorded, and
(2) a diffusion model is trained to produce the next frame, conditioned on the
sequence of past frames and actions. Conditioning augmentations enable stable
auto-regressive generation over long trajectories.
66ce94d1d3d85430b7ce06ad
Dani Valevski
admin_assigned
2024-08-28T08:03:06.126Z
632b5adf6110e37dba3e3ca3
/avatars/2c59a399d9b5d01d116fcb7775761fc8.svg
Dani Valevski
user
daniva
66ce94d1d3d85430b7ce06ae
Yaniv Leviathan
admin_assigned
2024-08-28T08:03:12.296Z
6394a2244db1a9bedd1f2229
/avatars/f04496811eb6873781e2ba606a2f97f0.svg
Yaniv Leviathan
user
yanivle
66ce94d1d3d85430b7ce06af
Moab Arar
claimed_verified
2024-08-28T07:50:53.649Z
60defb17bb3e1aaa18b005bb
/avatars/086264e93d6f7ba2eab5cc1b218df4a5.svg
Moab Arar
user
moabarar
66ce94d1d3d85430b7ce06b0
Shlomi Fruchter
admin_assigned
2024-08-28T08:03:17.576Z
6335ae887fcbeb615b0f9924
/avatars/60fd9305ccc56f17d6007f781c54fe2b.svg
Shlomi Fruchter
user
shlomif
Diffusion Models Are Real-Time Game Engines
We present GameNGen, the first game engine powered entirely by a neural model
that enables real-time interaction with a complex environment over long
trajectories at high quality. GameNGen can interactively simulate the classic
game DOOM at over 20 frames per second on a single TPU. Next frame prediction
achieves a PSNR of 29.4, comparable to lossy JPEG compression. Human raters are
only slightly better than random chance at distinguishing short clips of the
game from clips of the simulation. GameNGen is trained in two phases: (1) an
RL-agent learns to play the game and the training sessions are recorded, and
(2) a diffusion model is trained to produce the next frame, conditioned on the
sequence of past frames and actions. Conditioning augmentations enable stable
auto-regressive generation over long trajectories.
GameNGen, a neural model-powered game engine, simulates high-quality gameplay in real-time using a diffusion model conditioned on past frames and actions.
neural model real-time interaction complex environment long trajectories GameNGen classic game DOOM frames per second TPU next frame prediction PSNR _human raters RL-agent training sessions diffusion model conditioning augmentations stable auto-regressive generation