V-JEPA 2: Self-Supervised Video Models Enable Understanding, Prediction
  and Planning
A major challenge for modern AI is to learn to understand the world and learn
to act largely by observation. This paper explores a self-supervised approach
that combines internet-scale video data with a small amount of interaction data
(robot trajectories), to develop models capable of understanding, predicting,
and planning in the physical world. We first pre-train an action-free
joint-embedding-predictive architecture, V-JEPA 2, on a video and image dataset
comprising over 1 million hours of internet video. V-JEPA 2 achieves strong
performance on motion understanding (77.3 top-1 accuracy on Something-Something
v2) and state-of-the-art performance on human action anticipation (39.7
recall-at-5 on Epic-Kitchens-100) surpassing previous task-specific models.
Additionally, after aligning V-JEPA 2 with a large language model, we
demonstrate state-of-the-art performance on multiple video question-answering
tasks at the 8 billion parameter scale (e.g., 84.0 on PerceptionTest, 76.9 on
TempCompass). Finally, we show how self-supervised learning can be applied to
robotic planning tasks by post-training a latent action-conditioned world
model, V-JEPA 2-AC, using less than 62 hours of unlabeled robot videos from the
Droid dataset. We deploy V-JEPA 2-AC zero-shot on Franka arms in two different
labs and enable picking and placing of objects using planning with image goals.
Notably, this is achieved without collecting any data from the robots in these
environments, and without any task-specific training or reward. This work
demonstrates how self-supervised learning from web-scale data and a small
amount of robot interaction data can yield a world model capable of planning in
the physical world.
684b22c63b733ba333686eed
Mido Assran
684b22c63b733ba333686eee
Adrien Bardes
684b22c63b733ba333686eef
David Fan
684b22c63b733ba333686ef0
Quentin Garrido
684b22c63b733ba333686ef1
Russell Howes
684b22c63b733ba333686ef2
Mojtaba
684b22c63b733ba333686ef3
Komeili
684b22c63b733ba333686ef4
Matthew Muckley
684b22c63b733ba333686ef5
Ammar Rizvi
684b22c63b733ba333686ef6
Claire Roberts
684b22c63b733ba333686ef7
Koustuv Sinha
claimed_verified
2025-06-24T08:10:51.618Z
622b93067b9143726fbedc37
https://cdn-avatars.huggingface.co/v1/production/uploads/1647023505150-622b93067b9143726fbedc37.jpeg
Koustuv Sinha
user
koustuvs
684b22c63b733ba333686ef8
Artem Zholus
684b22c63b733ba333686ef9
Sergio Arnaud
684b22c63b733ba333686efa
Abha Gejji
684b22c63b733ba333686efb
Ada Martin
684b22c63b733ba333686efc
Francois Robert Hogan
684b22c63b733ba333686efd
Daniel Dugas
684b22c63b733ba333686efe
Piotr Bojanowski
684b22c63b733ba333686eff
Vasil Khalidov
684b22c63b733ba333686f00
Patrick Labatut
684b22c63b733ba333686f01
Francisco Massa
684b22c63b733ba333686f02
Marc Szafraniec
684b22c63b733ba333686f03
Kapil Krishnakumar
684b22c63b733ba333686f04
Yong Li
684b22c63b733ba333686f05
Xiaodong Ma
684b22c63b733ba333686f06
Sarath Chandar
684b22c63b733ba333686f07
Franziska Meier
684b22c63b733ba333686f08
Yann LeCun
684b22c63b733ba333686f09
Michael Rabbat
684b22c63b733ba333686f0a
Nicolas Ballas
V-JEPA 2: Self-Supervised Video Models Enable Understanding, Prediction
  and Planning
A major challenge for modern AI is to learn to understand the world and learn
to act largely by observation. This paper explores a self-supervised approach
that combines internet-scale video data with a small amount of interaction data
(robot trajectories), to develop models capable of understanding, predicting,
and planning in the physical world. We first pre-train an action-free
joint-embedding-predictive architecture, V-JEPA 2, on a video and image dataset
comprising over 1 million hours of internet video. V-JEPA 2 achieves strong
performance on motion understanding (77.3 top-1 accuracy on Something-Something
v2) and state-of-the-art performance on human action anticipation (39.7
recall-at-5 on Epic-Kitchens-100) surpassing previous task-specific models.
Additionally, after aligning V-JEPA 2 with a large language model, we
demonstrate state-of-the-art performance on multiple video question-answering
tasks at the 8 billion parameter scale (e.g., 84.0 on PerceptionTest, 76.9 on
TempCompass). Finally, we show how self-supervised learning can be applied to
robotic planning tasks by post-training a latent action-conditioned world
model, V-JEPA 2-AC, using less than 62 hours of unlabeled robot videos from the
Droid dataset. We deploy V-JEPA 2-AC zero-shot on Franka arms in two different
labs and enable picking and placing of objects using planning with image goals.
Notably, this is achieved without collecting any data from the robots in these
environments, and without any task-specific training or reward. This work
demonstrates how self-supervised learning from web-scale data and a small
amount of robot interaction data can yield a world model capable of planning in
the physical world.
A self-supervised approach combining internet video data and minimal robot interaction achieves strong performances in motion understanding, action anticipation, video question-answering, and robotic planning without task-specific training or reward.
self-supervised learning joint-embedding-predictive architecture V-JEPA 2 motion understanding human action anticipation video question-answering latent action-conditioned world model V-JEPA 2-AC robotic planning zero-shot deployment Franka arms image goals