Self-Supervised Learning of Motion Concepts by Optimizing
  Counterfactuals
Estimating motion in videos is an essential computer vision problem with many
downstream applications, including controllable video generation and robotics.
Current solutions are primarily trained using synthetic data or require tuning
of situation-specific heuristics, which inherently limits these models'
capabilities in real-world contexts. Despite recent developments in large-scale
self-supervised learning from videos, leveraging such representations for
motion estimation remains relatively underexplored. In this work, we develop
Opt-CWM, a self-supervised technique for flow and occlusion estimation from a
pre-trained next-frame prediction model. Opt-CWM works by learning to optimize
counterfactual probes that extract motion information from a base video model,
avoiding the need for fixed heuristics while training on unrestricted video
inputs. We achieve state-of-the-art performance for motion estimation on
real-world videos while requiring no labeled data.
67e4b46cc90e5edf25f581f8
Stefan Stojanov
67e4b46cc90e5edf25f581f9
David Wendt
admin_assigned
2025-03-27T10:14:24.479Z
6303e4bb0547362a22a62274
/avatars/91c9922913e45410ceed77f2e4971588.svg
David Wendt
user
kmeisthax
67e4b46cc90e5edf25f581fa
Seungwoo Kim
claimed_verified
2025-07-16T08:16:29.323Z
6876bd84da0a8020fe37880b
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/E3nTXyneLmVlOqFDyRkin.png
Seungwoo (Simon) Kim
user
SeKim12
67e4b46cc90e5edf25f581fb
Rahul Venkatesh
67e4b46cc90e5edf25f581fc
Kevin Feigelis
67e4b46cc90e5edf25f581fd
Jiajun Wu
67e4b46cc90e5edf25f581fe
Daniel LK Yamins
claimed_verified
2025-04-09T09:48:22.969Z
67f625c5f7a398c2a285d1b8
/avatars/1e4b708d68caee4a2d9ae6fe6e1c7b91.svg
Dan Yamins
user
dyamins
Self-Supervised Learning of Motion Concepts by Optimizing
  Counterfactuals
Estimating motion in videos is an essential computer vision problem with many
downstream applications, including controllable video generation and robotics.
Current solutions are primarily trained using synthetic data or require tuning
of situation-specific heuristics, which inherently limits these models'
capabilities in real-world contexts. Despite recent developments in large-scale
self-supervised learning from videos, leveraging such representations for
motion estimation remains relatively underexplored. In this work, we develop
Opt-CWM, a self-supervised technique for flow and occlusion estimation from a
pre-trained next-frame prediction model. Opt-CWM works by learning to optimize
counterfactual probes that extract motion information from a base video model,
avoiding the need for fixed heuristics while training on unrestricted video
inputs. We achieve state-of-the-art performance for motion estimation on
real-world videos while requiring no labeled data.
A self-supervised technique, Opt-CWM, extracts motion and occlusion information from pre-trained models without labeled data, achieving state-of-the-art performance on real-world videos.
self-supervised learning flow estimation occlusion estimation next-frame prediction counterfactual probes