Planning with Reasoning using Vision Language World Model
Effective planning requires strong world models, but high-level world models
that can understand and reason about actions with semantic and temporal
abstraction remain largely underdeveloped. We introduce the Vision Language
World Model (VLWM), a foundation model trained for language-based world
modeling on natural videos. Given visual observations, the VLWM first infers
the overall goal achievements then predicts a trajectory composed of
interleaved actions and world state changes. Those targets are extracted by
iterative LLM Self-Refine conditioned on compressed future observations
represented by Tree of Captions. The VLWM learns both an action policy and a
dynamics model, which respectively facilitates reactive system-1 plan decoding
and reflective system-2 planning via cost minimization. The cost evaluates the
semantic distance between the hypothetical future states given by VLWM
roll-outs and the expected goal state, and is measured by a critic model that
we trained in a self-supervised manner. The VLWM achieves state-of-the-art
Visual Planning for Assistance (VPA) performance on both benchmark evaluations
and our proposed PlannerArena human evaluations, where system-2 improves the
Elo score by +27% upon system-1. The VLWM models also outperforms strong VLM
baselines on RoboVQA and WorldPrediction benchmark.
68b9940c736018af705e8d58
Delong Chen
admin_assigned
2025-09-04T19:39:22.774Z
630491107424d937fa3258be
/avatars/b8bd81bc8544674ee26b78702afdb87c.svg
Delong Chen
user
chendelong
68b9940c736018af705e8d59
Theo Moutakanni
admin_assigned
2025-09-04T19:39:32.391Z
639111acb73a0f8c02a92844
/avatars/58766701471be66fd784bedd319741a4.svg
Theo Moutakanni
user
TheoM
68b9940c736018af705e8d5a
Willy Chung
admin_assigned
2025-09-04T19:39:39.541Z
6749afd958bd26b6c821d336
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/4idAccbPGO2okny1ru4RQ.png
Willy Chung
user
whcchung
68b9940c736018af705e8d5b
Yejin Bang
68b9940c736018af705e8d5c
Ziwei Ji
68b9940c736018af705e8d5d
Allen Bolourchi
admin_assigned
2025-09-04T19:39:45.340Z
65f08f0bcd30f1ed6ea6cbd1
/avatars/5eb2c2032dc2054d4ca86665ba0d428c.svg
Allen Bolourchi
user
allen-ml
68b9940c736018af705e8d5e
Pascale Fung
Planning with Reasoning using Vision Language World Model
Effective planning requires strong world models, but high-level world models
that can understand and reason about actions with semantic and temporal
abstraction remain largely underdeveloped. We introduce the Vision Language
World Model (VLWM), a foundation model trained for language-based world
modeling on natural videos. Given visual observations, the VLWM first infers
the overall goal achievements then predicts a trajectory composed of
interleaved actions and world state changes. Those targets are extracted by
iterative LLM Self-Refine conditioned on compressed future observations
represented by Tree of Captions. The VLWM learns both an action policy and a
dynamics model, which respectively facilitates reactive system-1 plan decoding
and reflective system-2 planning via cost minimization. The cost evaluates the
semantic distance between the hypothetical future states given by VLWM
roll-outs and the expected goal state, and is measured by a critic model that
we trained in a self-supervised manner. The VLWM achieves state-of-the-art
Visual Planning for Assistance (VPA) performance on both benchmark evaluations
and our proposed PlannerArena human evaluations, where system-2 improves the
Elo score by +27% upon system-1. The VLWM models also outperforms strong VLM
baselines on RoboVQA and WorldPrediction benchmark.
The Vision Language World Model (VLWM) achieves state-of-the-art performance in visual planning by integrating language-based world modeling, action policy learning, and dynamics modeling with semantic and temporal abstraction.
Vision Language World Model VLWM language-based world modeling natural videos iterative LLM Self-Refine Tree of Captions action policy dynamics model system-1 plan decoding system-2 planning cost minimization semantic distance critic model self-supervised manner Visual Planning for Assistance VPA PlannerArena Elo score RoboVQA WorldPrediction benchmark