High-Fidelity Simulated Data Generation for Real-World Zero-Shot Robotic
  Manipulation Learning with Gaussian Splatting
The scalability of robotic learning is fundamentally bottlenecked by the
significant cost and labor of real-world data collection. While simulated data
offers a scalable alternative, it often fails to generalize to the real world
due to significant gaps in visual appearance, physical properties, and object
interactions. To address this, we propose RoboSimGS, a novel Real2Sim2Real
framework that converts multi-view real-world images into scalable,
high-fidelity, and physically interactive simulation environments for robotic
manipulation. Our approach reconstructs scenes using a hybrid representation:
3D Gaussian Splatting (3DGS) captures the photorealistic appearance of the
environment, while mesh primitives for interactive objects ensure accurate
physics simulation. Crucially, we pioneer the use of a Multi-modal Large
Language Model (MLLM) to automate the creation of physically plausible,
articulated assets. The MLLM analyzes visual data to infer not only physical
properties (e.g., density, stiffness) but also complex kinematic structures
(e.g., hinges, sliding rails) of objects. We demonstrate that policies trained
entirely on data generated by RoboSimGS achieve successful zero-shot
sim-to-real transfer across a diverse set of real-world manipulation tasks.
Furthermore, data from RoboSimGS significantly enhances the performance and
generalization capabilities of SOTA methods. Our results validate RoboSimGS as
a powerful and scalable solution for bridging the sim-to-real gap.
68edb788de1fee572713a7ac
Haoyu Zhao
68edb788de1fee572713a7ad
Cheng Zeng
68edb788de1fee572713a7ae
Linghao Zhuang
68edb788de1fee572713a7af
Yaxi Zhao
68edb788de1fee572713a7b0
Shengke Xue
68edb788de1fee572713a7b1
Hao Wang
68edb788de1fee572713a7b2
Xingyue Zhao
68edb788de1fee572713a7b3
Zhongyu Li
68edb788de1fee572713a7b4
Kehan Li
68edb788de1fee572713a7b5
Siteng Huang
68edb788de1fee572713a7b6
Mingxiu Chen
68edb788de1fee572713a7b7
Xin Li
68edb788de1fee572713a7b8
Deli Zhao
68edb788de1fee572713a7b9
Hua Zou
High-Fidelity Simulated Data Generation for Real-World Zero-Shot Robotic
  Manipulation Learning with Gaussian Splatting
The scalability of robotic learning is fundamentally bottlenecked by the
significant cost and labor of real-world data collection. While simulated data
offers a scalable alternative, it often fails to generalize to the real world
due to significant gaps in visual appearance, physical properties, and object
interactions. To address this, we propose RoboSimGS, a novel Real2Sim2Real
framework that converts multi-view real-world images into scalable,
high-fidelity, and physically interactive simulation environments for robotic
manipulation. Our approach reconstructs scenes using a hybrid representation:
3D Gaussian Splatting (3DGS) captures the photorealistic appearance of the
environment, while mesh primitives for interactive objects ensure accurate
physics simulation. Crucially, we pioneer the use of a Multi-modal Large
Language Model (MLLM) to automate the creation of physically plausible,
articulated assets. The MLLM analyzes visual data to infer not only physical
properties (e.g., density, stiffness) but also complex kinematic structures
(e.g., hinges, sliding rails) of objects. We demonstrate that policies trained
entirely on data generated by RoboSimGS achieve successful zero-shot
sim-to-real transfer across a diverse set of real-world manipulation tasks.
Furthermore, data from RoboSimGS significantly enhances the performance and
generalization capabilities of SOTA methods. Our results validate RoboSimGS as
a powerful and scalable solution for bridging the sim-to-real gap.
RoboSimGS, a Real2Sim2Real framework, uses 3D Gaussian Splatting and mesh primitives to create scalable, high-fidelity, and physically interactive simulation environments, enabling successful zero-shot sim-to-real transfer for robotic manipulation tasks.
3D Gaussian Splatting mesh primitives Multi-modal Large Language Model physically plausible articulated assets zero-shot sim-to-real transfer robotic manipulation