Manipulation as in Simulation: Enabling Accurate Geometry Perception in
  Robots
Modern robotic manipulation primarily relies on visual observations in a 2D
color space for skill learning but suffers from poor generalization. In
contrast, humans, living in a 3D world, depend more on physical properties-such
as distance, size, and shape-than on texture when interacting with objects.
Since such 3D geometric information can be acquired from widely available depth
cameras, it appears feasible to endow robots with similar perceptual
capabilities. Our pilot study found that using depth cameras for manipulation
is challenging, primarily due to their limited accuracy and susceptibility to
various types of noise. In this work, we propose Camera Depth Models (CDMs) as
a simple plugin on daily-use depth cameras, which take RGB images and raw depth
signals as input and output denoised, accurate metric depth. To achieve this,
we develop a neural data engine that generates high-quality paired data from
simulation by modeling a depth camera's noise pattern. Our results show that
CDMs achieve nearly simulation-level accuracy in depth prediction, effectively
bridging the sim-to-real gap for manipulation tasks. Notably, our experiments
demonstrate, for the first time, that a policy trained on raw simulated depth,
without the need for adding noise or real-world fine-tuning, generalizes
seamlessly to real-world robots on two challenging long-horizon tasks involving
articulated, reflective, and slender objects, with little to no performance
degradation. We hope our findings will inspire future research in utilizing
simulation data and 3D information in general robot policies.
68b9eccf736018af705e8ddf
Minghuan Liu
admin_assigned
2025-09-04T20:12:43.594Z
630c8f0c8e3ff0c72327140a
/avatars/04a5f2effd0ff25d9fb531b195fd8846.svg
Minghuan Liu
user
ericonaldo
68b9eccf736018af705e8de0
Zhengbang Zhu
admin_assigned
2025-09-04T20:12:49.069Z
64ec1840c2bcaa4525eabc20
/avatars/bcd5be2ccf6527292d66d0a985ff663f.svg
Zhengbang Zhu
user
Avada11
68b9eccf736018af705e8de1
Xiaoshen Han
admin_assigned
2025-09-04T20:12:54.694Z
64256c3bdaa3502ee357380c
/avatars/d47a0268c4c3366cbfb4d621017e9b1e.svg
Xiaoshen Han
user
xshenhan
68b9eccf736018af705e8de2
Peng Hu
68b9eccf736018af705e8de3
Haotong Lin
68b9eccf736018af705e8de4
Xinyao Li
68b9eccf736018af705e8de5
Jingxiao Chen
admin_assigned
2025-09-04T20:13:08.421Z
67b477a81ae54de19c087823
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/T7r-L7FNVKnymSo7kYqqX.png
Jingxiao Chen
user
TimerChen
68b9eccf736018af705e8de6
Jiafeng Xu
68b9eccf736018af705e8de7
Yichu Yang
68b9eccf736018af705e8de8
Yunfeng Lin
admin_assigned
2025-09-04T20:13:20.218Z
6889daab799265067d8aeedf
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/b-CqqzpCDqg7SgIRrhXkk.png
YunFeng_Lin
user
Edward0000
68b9eccf736018af705e8de9
Xinghang Li
admin_assigned
2025-09-04T20:13:26.028Z
646c907297819a8be93a2aad
/avatars/b76734ab89d8446e12a5b679716aee78.svg
Xinghang Li
user
hywslxh
68b9eccf736018af705e8dea
Yong Yu
68b9eccf736018af705e8deb
Weinan Zhang
68b9eccf736018af705e8dec
Tao Kong
68b9eccf736018af705e8ded
Bingyi Kang
admin_assigned
2025-09-04T20:12:37.779Z
647b5fef6a79fbf5e996c47c
https://cdn-avatars.huggingface.co/v1/production/uploads/647b5fef6a79fbf5e996c47c/IkSMnDsCY_CyEFCiMDuxe.jpeg
Bingyi Kang
user
bykang
Manipulation as in Simulation: Enabling Accurate Geometry Perception in
  Robots
Modern robotic manipulation primarily relies on visual observations in a 2D
color space for skill learning but suffers from poor generalization. In
contrast, humans, living in a 3D world, depend more on physical properties-such
as distance, size, and shape-than on texture when interacting with objects.
Since such 3D geometric information can be acquired from widely available depth
cameras, it appears feasible to endow robots with similar perceptual
capabilities. Our pilot study found that using depth cameras for manipulation
is challenging, primarily due to their limited accuracy and susceptibility to
various types of noise. In this work, we propose Camera Depth Models (CDMs) as
a simple plugin on daily-use depth cameras, which take RGB images and raw depth
signals as input and output denoised, accurate metric depth. To achieve this,
we develop a neural data engine that generates high-quality paired data from
simulation by modeling a depth camera's noise pattern. Our results show that
CDMs achieve nearly simulation-level accuracy in depth prediction, effectively
bridging the sim-to-real gap for manipulation tasks. Notably, our experiments
demonstrate, for the first time, that a policy trained on raw simulated depth,
without the need for adding noise or real-world fine-tuning, generalizes
seamlessly to real-world robots on two challenging long-horizon tasks involving
articulated, reflective, and slender objects, with little to no performance
degradation. We hope our findings will inspire future research in utilizing
simulation data and 3D information in general robot policies.
Camera Depth Models (CDMs) enhance depth camera accuracy by denoising and improving metric depth prediction, enabling better generalization of robotic manipulation policies from simulation to real-world tasks.
Camera Depth Models neural data engine depth camera noise pattern depth prediction sim-to-real gap robotic manipulation articulated objects reflective objects slender objects