Taming generative world models for zero-shot optical flow extraction
Seungwoo Kim, Khai Loong Aw, Klemen Kotar, Cristobal Eyzaguirre, Wanhee Lee, Yunong Liu, Jared Watrous, Stefan Stojanov, Juan Carlos Niebles, Jiajun Wu, Daniel Yamins
Extracting dense motion (optical flow) from videos remains a core computer-vision problem. Motivated by the recent success of large general-purpose models, we ask whether frozen self-supervised video world models trained only to predict future frames can be prompted, without fine-tuning, to output flow. Prior attempts to read out depth or illumination from video generators required fine-tuning; that strategy is ill-suited for flow, where labeled data are scarce and synthetic datasets suffer from a sim-to-real gap. We study several popular generative model architectures and find that successful zero-shot flow extraction requires three model properties: (1) distributional prediction of future frames (avoiding blurry or noisy outputs); (2) factorized latents that treat each spatio-temporal patch independently; and (3) random-access decoding that can condition on any subset of future pixels. These criteria are met by the recently introduced Local Random Access Sequence (LRAS) architecture. Building on LRAS, we propose KL-tracing: a procedure for injecting a small, local perturbation into the first frame, rolling out the model one step, and computing the Kullback–Leibler divergence between perturbed and unperturbed predictive distributions. The KL peak traces the displacement field, yielding optical flow in a single forward pass. Our method outperforms state-of-the-art models on real-world TAP-Vid DAVIS dataset (16.6% relative improvement for endpoint error) and synthetic TAP-Vid Kubric (4.7% relative improvement), despite being trained on real-world videos. Our results indicate that prompting controllable, self-supervised world models is a scalable and effective alternative to supervised or photometric-loss approaches for high-quality optical flow.