World Modeling with Probabilistic Structure Integration
We present Probabilistic Structure Integration (PSI), a system for learning
richly controllable and flexibly promptable world models from data. PSI
consists of a three-step cycle. The first step, Probabilistic prediction,
involves building a probabilistic graphical model Psi of the data, in the form
of a random-access autoregressive sequence model. Psi supports a complete set
of learned conditional distributions describing the dependence of any variables
in the data on any other set of variables. In step 2, Structure extraction, we
show how to extract underlying low-dimensional properties in the data,
corresponding to a diverse set of meaningful "intermediate structures", in a
zero-shot fashion via causal inference on Psi. Step 3, Integration, completes
the cycle by converting these structures into new token types that are then
continually mixed back into the training diet as conditioning signals and
prediction targets. Each such cycle augments the capabilities of Psi, both
allowing it to model the underlying data better, and creating new control
handles -- akin to an LLM-like universal prompting language. We train an
instance of Psi on 1.4 trillion tokens of internet video data; we use it to
perform a variety of useful video prediction and understanding inferences; we
extract state-of-the-art optical flow, self-supervised depth and object
segmentation; and we use these structures to support a full cycle of predictive
improvements.
68c848dd733e345e52ac1dd1
Klemen Kotar
claimed_verified
2025-09-16T09:42:59.207Z
646d3cae4220471ca0c6cb13
/avatars/30e47db03a23d49d946cc5248b28959a.svg
Klemen Kotar
user
klemenk
68c848dd733e345e52ac1dd2
Wanhee Lee
68c848dd733e345e52ac1dd3
Rahul Venkatesh
68c848dd733e345e52ac1dd4
Honglin Chen
68c848dd733e345e52ac1dd5
Daniel Bear
68c848dd733e345e52ac1dd6
Jared Watrous
68c848dd733e345e52ac1dd7
Simon Kim
68c848dd733e345e52ac1dd8
Khai Loong Aw
68c848dd733e345e52ac1dd9
Lilian Naing Chen
68c848dd733e345e52ac1dda
Stefan Stojanov
68c848dd733e345e52ac1ddb
Kevin Feigelis
68c848dd733e345e52ac1ddc
Imran Thobani
68c848dd733e345e52ac1ddd
Alex Durango
68c848dd733e345e52ac1dde
Khaled Jedoui
68c848dd733e345e52ac1ddf
Atlas Kazemian
68c848dd733e345e52ac1de0
Dan Yamins
World Modeling with Probabilistic Structure Integration
We present Probabilistic Structure Integration (PSI), a system for learning
richly controllable and flexibly promptable world models from data. PSI
consists of a three-step cycle. The first step, Probabilistic prediction,
involves building a probabilistic graphical model Psi of the data, in the form
of a random-access autoregressive sequence model. Psi supports a complete set
of learned conditional distributions describing the dependence of any variables
in the data on any other set of variables. In step 2, Structure extraction, we
show how to extract underlying low-dimensional properties in the data,
corresponding to a diverse set of meaningful "intermediate structures", in a
zero-shot fashion via causal inference on Psi. Step 3, Integration, completes
the cycle by converting these structures into new token types that are then
continually mixed back into the training diet as conditioning signals and
prediction targets. Each such cycle augments the capabilities of Psi, both
allowing it to model the underlying data better, and creating new control
handles -- akin to an LLM-like universal prompting language. We train an
instance of Psi on 1.4 trillion tokens of internet video data; we use it to
perform a variety of useful video prediction and understanding inferences; we
extract state-of-the-art optical flow, self-supervised depth and object
segmentation; and we use these structures to support a full cycle of predictive
improvements.
Probabilistic Structure Integration (PSI) learns richly controllable world models from data through probabilistic prediction, structure extraction, and integration, enhancing video prediction and understanding.
probabilistic graphical model random-access autoregressive sequence model conditional distributions causal inference intermediate structures token types predictive improvements optical flow self-supervised depth object segmentation