Neural feels with neural fields: Visuo-tactile perception for in-hand
  manipulation
To achieve human-level dexterity, robots must infer spatial awareness from
multimodal sensing to reason over contact interactions. During in-hand
manipulation of novel objects, such spatial awareness involves estimating the
object's pose and shape. The status quo for in-hand perception primarily
employs vision, and restricts to tracking a priori known objects. Moreover,
visual occlusion of objects in-hand is imminent during manipulation, preventing
current systems to push beyond tasks without occlusion. We combine vision and
touch sensing on a multi-fingered hand to estimate an object's pose and shape
during in-hand manipulation. Our method, NeuralFeels, encodes object geometry
by learning a neural field online and jointly tracks it by optimizing a pose
graph problem. We study multimodal in-hand perception in simulation and the
real-world, interacting with different objects via a proprioception-driven
policy. Our experiments show final reconstruction F-scores of 81% and average
pose drifts of 4.7,mm, further reduced to 2.3,mm with known
CAD models. Additionally, we observe that under heavy visual occlusion we can
achieve up to 94% improvements in tracking compared to vision-only methods.
Our results demonstrate that touch, at the very least, refines and, at the very
best, disambiguates visual estimates during in-hand manipulation. We release
our evaluation dataset of 70 experiments, FeelSight, as a step towards
benchmarking in this domain. Our neural representation driven by multimodal
sensing can serve as a perception backbone towards advancing robot dexterity.
Videos can be found on our project website
https://suddhu.github.io/neural-feels/
6584ecd0f8fae1c1e5b0927f
Sudharshan Suresh
admin_assigned
2023-12-22T12:39:14.099Z
645ba7079522edfc8e4fbf19
/avatars/888e3219d7591accc3a23bba8532273e.svg
Sudharshan Suresh
user
suddhu
6584ecd0f8fae1c1e5b09280
Haozhi Qi
claimed_verified
2023-12-26T10:37:57.135Z
658871e7d1331d552be8b457
/avatars/c323178ad1e0fbdfa5e66dcd4ba357e6.svg
Haozhi Qi
user
HaozhiQi
6584ecd0f8fae1c1e5b09281
Tingfan Wu
6584ecd0f8fae1c1e5b09282
Taosha Fan
6584ecd0f8fae1c1e5b09283
Luis Pineda
6584ecd0f8fae1c1e5b09284
Mike Lambeta
6584ecd0f8fae1c1e5b09285
Jitendra Malik
admin_assigned
2023-12-22T12:43:13.318Z
65369a95605a07338de78ab0
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/sGFjOjLT2akN-sn5beVWL.jpeg
Jitendra Malik 
user
jitendra1995
6584ecd0f8fae1c1e5b09286
Mrinal Kalakrishnan
admin_assigned
2023-12-22T12:42:53.762Z
64cad11dead94891d1e72243
/avatars/d7faa419ccb0182ad081555491b80d14.svg
Mrinal Kalakrishnan
user
kalakris
6584ecd0f8fae1c1e5b09287
Roberto Calandra
6584ecd0f8fae1c1e5b09288
Michael Kaess
6584ecd0f8fae1c1e5b09289
Joseph Ortiz
6584ecd0f8fae1c1e5b0928a
Mustafa Mukadam
claimed_verified
2024-01-02T09:28:09.480Z
65901cd989f1ff04636c896c
/avatars/64d1d58811f31d9304287699e0879d4b.svg
Mustafa Mustafa
user
mukadammh
Neural feels with neural fields: Visuo-tactile perception for in-hand
  manipulation
To achieve human-level dexterity, robots must infer spatial awareness from
multimodal sensing to reason over contact interactions. During in-hand
manipulation of novel objects, such spatial awareness involves estimating the
object's pose and shape. The status quo for in-hand perception primarily
employs vision, and restricts to tracking a priori known objects. Moreover,
visual occlusion of objects in-hand is imminent during manipulation, preventing
current systems to push beyond tasks without occlusion. We combine vision and
touch sensing on a multi-fingered hand to estimate an object's pose and shape
during in-hand manipulation. Our method, NeuralFeels, encodes object geometry
by learning a neural field online and jointly tracks it by optimizing a pose
graph problem. We study multimodal in-hand perception in simulation and the
real-world, interacting with different objects via a proprioception-driven
policy. Our experiments show final reconstruction F-scores of 81% and average
pose drifts of 4.7,mm, further reduced to 2.3,mm with known
CAD models. Additionally, we observe that under heavy visual occlusion we can
achieve up to 94% improvements in tracking compared to vision-only methods.
Our results demonstrate that touch, at the very least, refines and, at the very
best, disambiguates visual estimates during in-hand manipulation. We release
our evaluation dataset of 70 experiments, FeelSight, as a step towards
benchmarking in this domain. Our neural representation driven by multimodal
sensing can serve as a perception backbone towards advancing robot dexterity.
Videos can be found on our project website
https://suddhu.github.io/neural-feels/
NeuralFeels, a multimodal perception method combining vision and touch, enhances spatial awareness and object tracking during in-hand manipulation, especially under visual occlusion.
neural field pose graph problem proprioception-driven policy F-scores pose drifts visual occlusion multimodal sensing perception backbone