MineWorld: a Real-Time and Open-Source Interactive World Model on
  Minecraft
World modeling is a crucial task for enabling intelligent agents to
effectively interact with humans and operate in dynamic environments. In this
work, we propose MineWorld, a real-time interactive world model on Minecraft,
an open-ended sandbox game which has been utilized as a common testbed for
world modeling. MineWorld is driven by a visual-action autoregressive
Transformer, which takes paired game scenes and corresponding actions as input,
and generates consequent new scenes following the actions. Specifically, by
transforming visual game scenes and actions into discrete token ids with an
image tokenizer and an action tokenizer correspondingly, we consist the model
input with the concatenation of the two kinds of ids interleaved. The model is
then trained with next token prediction to learn rich representations of game
states as well as the conditions between states and actions simultaneously. In
inference, we develop a novel parallel decoding algorithm that predicts the
spatial redundant tokens in each frame at the same time, letting models in
different scales generate 4 to 7 frames per second and enabling real-time
interactions with game players. In evaluation, we propose new metrics to assess
not only visual quality but also the action following capacity when generating
new scenes, which is crucial for a world model. Our comprehensive evaluation
shows the efficacy of MineWorld, outperforming SoTA open-sourced diffusion
based world models significantly. The code and model have been released.
67fc7367df5f5d1e87c14c6a
Junliang Guo
67fc7367df5f5d1e87c14c6b
Yang Ye
67fc7367df5f5d1e87c14c6c
Tianyu He
67fc7367df5f5d1e87c14c6d
Haoyu Wu
claimed_verified
2025-07-22T07:54:50.058Z
6590f7880c993129053a2344
/avatars/d08049493234edb8e23f1c1531e386d3.svg
Haoyu wu
user
Haoyuwu
67fc7367df5f5d1e87c14c6e
Yushu Jiang
67fc7367df5f5d1e87c14c6f
Tim Pearce
67fc7367df5f5d1e87c14c70
Jiang Bian
claimed_verified
2025-06-19T09:10:59.200Z
683843c8a9bdf98f92657042
/avatars/4983876f615fa830cfc8f2cb5a00fa08.svg
Jiang Bian
user
jiangbian22
MineWorld: a Real-Time and Open-Source Interactive World Model on
  Minecraft
World modeling is a crucial task for enabling intelligent agents to
effectively interact with humans and operate in dynamic environments. In this
work, we propose MineWorld, a real-time interactive world model on Minecraft,
an open-ended sandbox game which has been utilized as a common testbed for
world modeling. MineWorld is driven by a visual-action autoregressive
Transformer, which takes paired game scenes and corresponding actions as input,
and generates consequent new scenes following the actions. Specifically, by
transforming visual game scenes and actions into discrete token ids with an
image tokenizer and an action tokenizer correspondingly, we consist the model
input with the concatenation of the two kinds of ids interleaved. The model is
then trained with next token prediction to learn rich representations of game
states as well as the conditions between states and actions simultaneously. In
inference, we develop a novel parallel decoding algorithm that predicts the
spatial redundant tokens in each frame at the same time, letting models in
different scales generate 4 to 7 frames per second and enabling real-time
interactions with game players. In evaluation, we propose new metrics to assess
not only visual quality but also the action following capacity when generating
new scenes, which is crucial for a world model. Our comprehensive evaluation
shows the efficacy of MineWorld, outperforming SoTA open-sourced diffusion
based world models significantly. The code and model have been released.
MineWorld, a real-time interactive world model built on Minecraft, uses a visual-action autoregressive Transformer to generate consequent game scenes based on inputs of game scenes and actions, outperforming state-of-the-art diffusion-based models.
visual-action autoregressive Transformer image tokenizer action tokenizer next token prediction parallel decoding algorithm visual quality action following capacity world modeling