X-Sim: Cross-Embodiment Learning via Real-to-Sim-to-Real
Human videos offer a scalable way to train robot manipulation policies, but
lack the action labels needed by standard imitation learning algorithms.
Existing cross-embodiment approaches try to map human motion to robot actions,
but often fail when the embodiments differ significantly. We propose X-Sim, a
real-to-sim-to-real framework that uses object motion as a dense and
transferable signal for learning robot policies. X-Sim starts by reconstructing
a photorealistic simulation from an RGBD human video and tracking object
trajectories to define object-centric rewards. These rewards are used to train
a reinforcement learning (RL) policy in simulation. The learned policy is then
distilled into an image-conditioned diffusion policy using synthetic rollouts
rendered with varied viewpoints and lighting. To transfer to the real world,
X-Sim introduces an online domain adaptation technique that aligns real and
simulated observations during deployment. Importantly, X-Sim does not require
any robot teleoperation data. We evaluate it across 5 manipulation tasks in 2
environments and show that it: (1) improves task progress by 30% on average
over hand-tracking and sim-to-real baselines, (2) matches behavior cloning with
10x less data collection time, and (3) generalizes to new camera viewpoints and
test-time changes. Code and videos are available at
https://portal-cornell.github.io/X-Sim/.
68276829c73b5a259249b459
Prithwish Dan
admin_assigned
2025-05-19T07:26:23.583Z
65e8c6cae214f37d85e341c0
https://cdn-avatars.huggingface.co/v1/production/uploads/65e8c6cae214f37d85e341c0/CtpYeBEqap9TwomncZI8V.png
Prithwish Dan
user
prithwishdan
68276829c73b5a259249b45a
Kushal Kedia
admin_assigned
2025-05-19T07:26:38.361Z
65e1ff68ec41b21de6201cbd
/avatars/498f8d0b5d83c18e9aefdf730f5be170.svg
Kushal Kedia
user
Kushal20
68276829c73b5a259249b45b
Angela Chao
admin_assigned
2025-05-19T07:26:45.371Z
65e3d60d00680b19bc015b0a
/avatars/f0d8e56a8181f0a59439ebe961ef5b06.svg
Angela Chao
user
ac2323
68276829c73b5a259249b45c
Edward Weiyi Duan
68276829c73b5a259249b45d
Maximus Adrian Pace
68276829c73b5a259249b45e
Wei-Chiu Ma
admin_assigned
2025-05-19T07:27:01.583Z
624fb0c8841b0a7f1b583e72
/avatars/a695a528817f57a98c946056d6f9b2f9.svg
Wei-Chiu Ma
user
weichium
68276829c73b5a259249b45f
Sanjiban Choudhury
X-Sim: Cross-Embodiment Learning via Real-to-Sim-to-Real
Human videos offer a scalable way to train robot manipulation policies, but
lack the action labels needed by standard imitation learning algorithms.
Existing cross-embodiment approaches try to map human motion to robot actions,
but often fail when the embodiments differ significantly. We propose X-Sim, a
real-to-sim-to-real framework that uses object motion as a dense and
transferable signal for learning robot policies. X-Sim starts by reconstructing
a photorealistic simulation from an RGBD human video and tracking object
trajectories to define object-centric rewards. These rewards are used to train
a reinforcement learning (RL) policy in simulation. The learned policy is then
distilled into an image-conditioned diffusion policy using synthetic rollouts
rendered with varied viewpoints and lighting. To transfer to the real world,
X-Sim introduces an online domain adaptation technique that aligns real and
simulated observations during deployment. Importantly, X-Sim does not require
any robot teleoperation data. We evaluate it across 5 manipulation tasks in 2
environments and show that it: (1) improves task progress by 30% on average
over hand-tracking and sim-to-real baselines, (2) matches behavior cloning with
10x less data collection time, and (3) generalizes to new camera viewpoints and
test-time changes. Code and videos are available at
https://portal-cornell.github.io/X-Sim/.
X-Sim uses photorealistic simulations and reinforcement learning to train robot manipulation policies from human videos without labeled actions, enhancing generalization and efficiency.
photorealistic simulation object-centric rewards reinforcement learning diffusion policy synthetic rollouts domain adaptation hand-tracking sim-to-real