Splatting Physical Scenes: End-to-End Real-to-Sim from Imperfect Robot
  Data
Creating accurate, physical simulations directly from real-world robot motion
holds great value for safe, scalable, and affordable robot learning, yet
remains exceptionally challenging. Real robot data suffers from occlusions,
noisy camera poses, dynamic scene elements, which hinder the creation of
geometrically accurate and photorealistic digital twins of unseen objects. We
introduce a novel real-to-sim framework tackling all these challenges at once.
Our key insight is a hybrid scene representation merging the photorealistic
rendering of 3D Gaussian Splatting with explicit object meshes suitable for
physics simulation within a single representation. We propose an end-to-end
optimization pipeline that leverages differentiable rendering and
differentiable physics within MuJoCo to jointly refine all scene components -
from object geometry and appearance to robot poses and physical parameters -
directly from raw and imprecise robot trajectories. This unified optimization
allows us to simultaneously achieve high-fidelity object mesh reconstruction,
generate photorealistic novel views, and perform annotation-free robot pose
calibration. We demonstrate the effectiveness of our approach both in
simulation and on challenging real-world sequences using an ALOHA 2 bi-manual
manipulator, enabling more practical and robust real-to-simulation pipelines.
6846b4453ec10bdd8ab4dbc9
Ben Moran
6846b4453ec10bdd8ab4dbca
Mauro Comi
claimed_verified
2025-06-10T09:21:51.578Z
62f6dd9b2e53c2efd33f8207
/avatars/887d795e0e650a8cc67e66f552187a73.svg
Mauro Comi
user
MauroC
6846b4453ec10bdd8ab4dbcb
Steven Bohez
6846b4453ec10bdd8ab4dbcc
Tom Erez
6846b4453ec10bdd8ab4dbcd
Zhibin Li
6846b4453ec10bdd8ab4dbce
Leonard Hasenclever
Splatting Physical Scenes: End-to-End Real-to-Sim from Imperfect Robot
  Data
Creating accurate, physical simulations directly from real-world robot motion
holds great value for safe, scalable, and affordable robot learning, yet
remains exceptionally challenging. Real robot data suffers from occlusions,
noisy camera poses, dynamic scene elements, which hinder the creation of
geometrically accurate and photorealistic digital twins of unseen objects. We
introduce a novel real-to-sim framework tackling all these challenges at once.
Our key insight is a hybrid scene representation merging the photorealistic
rendering of 3D Gaussian Splatting with explicit object meshes suitable for
physics simulation within a single representation. We propose an end-to-end
optimization pipeline that leverages differentiable rendering and
differentiable physics within MuJoCo to jointly refine all scene components -
from object geometry and appearance to robot poses and physical parameters -
directly from raw and imprecise robot trajectories. This unified optimization
allows us to simultaneously achieve high-fidelity object mesh reconstruction,
generate photorealistic novel views, and perform annotation-free robot pose
calibration. We demonstrate the effectiveness of our approach both in
simulation and on challenging real-world sequences using an ALOHA 2 bi-manual
manipulator, enabling more practical and robust real-to-simulation pipelines.
A novel real-to-sim framework merges 3D Gaussian Splatting and object meshes for accurate physics simulation, refining geometry, appearance, and robot poses from raw trajectories.
3D Gaussian Splatting object meshes physics simulation differentiable rendering differentiable physics MuJoCo scene reconstruction photorealistic rendering robot pose calibration ALOHA 2