Whole-Body Conditioned Egocentric Video Prediction
We train models to Predict Ego-centric Video from human Actions (PEVA), given
the past video and an action represented by the relative 3D body pose. By
conditioning on kinematic pose trajectories, structured by the joint hierarchy
of the body, our model learns to simulate how physical human actions shape the
environment from a first-person point of view. We train an auto-regressive
conditional diffusion transformer on Nymeria, a large-scale dataset of
real-world egocentric video and body pose capture. We further design a
hierarchical evaluation protocol with increasingly challenging tasks, enabling
a comprehensive analysis of the model's embodied prediction and control
abilities. Our work represents an initial attempt to tackle the challenges of
modeling complex real-world environments and embodied agent behaviors with
video prediction from the perspective of a human.
685e161b71131fa43be08b04
Yutong Bai
claimed_verified
2025-06-27T08:56:43.620Z
6332253749a95639154cc894
/avatars/f8e53fff5a324591026114c431cd407e.svg
Yutong Bai
user
Emma02
685e161b71131fa43be08b05
Danny Tran
claimed_verified
2025-06-27T08:56:45.405Z
658a1f4a35f23c0f1c4f689f
/avatars/e800fcbbcd242f311c3896a603862416.svg
Danny Tran
user
dans123
685e161b71131fa43be08b06
Amir Bar
685e161b71131fa43be08b07
Yann LeCun
685e161b71131fa43be08b08
Trevor Darrell
685e161b71131fa43be08b09
Jitendra Malik
Whole-Body Conditioned Egocentric Video Prediction
We train models to Predict Ego-centric Video from human Actions (PEVA), given
the past video and an action represented by the relative 3D body pose. By
conditioning on kinematic pose trajectories, structured by the joint hierarchy
of the body, our model learns to simulate how physical human actions shape the
environment from a first-person point of view. We train an auto-regressive
conditional diffusion transformer on Nymeria, a large-scale dataset of
real-world egocentric video and body pose capture. We further design a
hierarchical evaluation protocol with increasingly challenging tasks, enabling
a comprehensive analysis of the model's embodied prediction and control
abilities. Our work represents an initial attempt to tackle the challenges of
modeling complex real-world environments and embodied agent behaviors with
video prediction from the perspective of a human.
A model trained on real-world egocentric video and body pose predicts video from human actions using an auto-regressive conditional diffusion transformer, evaluated with a hierarchical protocol of tasks.
auto-regressive conditional diffusion transformer