DINO-WM: World Models on Pre-trained Visual Features enable Zero-shot
  Planning
The ability to predict future outcomes given control actions is fundamental
for physical reasoning. However, such predictive models, often called world
models, have proven challenging to learn and are typically developed for
task-specific solutions with online policy learning. We argue that the true
potential of world models lies in their ability to reason and plan across
diverse problems using only passive data. Concretely, we require world models
to have the following three properties: 1) be trainable on offline,
pre-collected trajectories, 2) support test-time behavior optimization, and 3)
facilitate task-agnostic reasoning. To realize this, we present DINO World
Model (DINO-WM), a new method to model visual dynamics without reconstructing
the visual world. DINO-WM leverages spatial patch features pre-trained with
DINOv2, enabling it to learn from offline behavioral trajectories by predicting
future patch features. This design allows DINO-WM to achieve observational
goals through action sequence optimization, facilitating task-agnostic behavior
planning by treating desired goal patch features as prediction targets. We
evaluate DINO-WM across various domains, including maze navigation, tabletop
pushing, and particle manipulation. Our experiments demonstrate that DINO-WM
can generate zero-shot behavioral solutions at test time without relying on
expert demonstrations, reward modeling, or pre-learned inverse models. Notably,
DINO-WM exhibits strong generalization capabilities compared to prior
state-of-the-art work, adapting to diverse task families such as arbitrarily
configured mazes, push manipulation with varied object shapes, and
multi-particle scenarios.
67a0783a1b24595484396c4d
Gaoyue Zhou
admin_assigned
2025-02-03T12:58:14.740Z
63486560e0bf88ccd36fe568
/avatars/934cffbd9f5c699abad20dcf86745382.svg
Gaoyue Zhou
user
gaoyuezhou
67a0783a1b24595484396c4e
Hengkai Pan
admin_assigned
2025-02-03T12:58:20.238Z
634236fc8d8089ebaefb8180
/avatars/e40518ff3f0d0a58ba4f46048c84640d.svg
Hengkai Pan
user
garyphk
67a0783a1b24595484396c4f
Yann LeCun
admin_assigned
2025-02-03T12:58:26.526Z
64ed0b8c2203a126eb1a5b9a
/avatars/9156dc406ed3f9ee62b73657ac20f5ed.svg
Yann LeCun
user
ylecun
67a0783a1b24595484396c50
Lerrel Pinto
admin_assigned
2025-02-03T12:58:41.008Z
66fffa0766bc54f4e532e3d2
/avatars/9db8b2183097bcaddded06d1b800cf77.svg
Lerrel Pinto
user
LerrelPinto
DINO-WM: World Models on Pre-trained Visual Features enable Zero-shot
  Planning
The ability to predict future outcomes given control actions is fundamental
for physical reasoning. However, such predictive models, often called world
models, have proven challenging to learn and are typically developed for
task-specific solutions with online policy learning. We argue that the true
potential of world models lies in their ability to reason and plan across
diverse problems using only passive data. Concretely, we require world models
to have the following three properties: 1) be trainable on offline,
pre-collected trajectories, 2) support test-time behavior optimization, and 3)
facilitate task-agnostic reasoning. To realize this, we present DINO World
Model (DINO-WM), a new method to model visual dynamics without reconstructing
the visual world. DINO-WM leverages spatial patch features pre-trained with
DINOv2, enabling it to learn from offline behavioral trajectories by predicting
future patch features. This design allows DINO-WM to achieve observational
goals through action sequence optimization, facilitating task-agnostic behavior
planning by treating desired goal patch features as prediction targets. We
evaluate DINO-WM across various domains, including maze navigation, tabletop
pushing, and particle manipulation. Our experiments demonstrate that DINO-WM
can generate zero-shot behavioral solutions at test time without relying on
expert demonstrations, reward modeling, or pre-learned inverse models. Notably,
DINO-WM exhibits strong generalization capabilities compared to prior
state-of-the-art work, adapting to diverse task families such as arbitrarily
configured mazes, push manipulation with varied object shapes, and
multi-particle scenarios.
DINO World Model learns from offline trajectories to predict future visual dynamics and optimize behavior for diverse tasks without expert demonstrations or reward modeling.
world models DINO World Model DINO-WM spatial patch features DINOv2 offline behavioral trajectories action sequence optimization task-agnostic behavior planning maze navigation tabletop pushing particle manipulation zero-shot behavioral solutions generalization capabilities