Learning Generalizable Feature Fields for Mobile Manipulation
An open problem in mobile manipulation is how to represent objects and scenes
in a unified manner, so that robots can use it both for navigating in the
environment and manipulating objects. The latter requires capturing intricate
geometry while understanding fine-grained semantics, whereas the former
involves capturing the complexity inherit to an expansive physical scale. In
this work, we present GeFF (Generalizable Feature Fields), a scene-level
generalizable neural feature field that acts as a unified representation for
both navigation and manipulation that performs in real-time. To do so, we treat
generative novel view synthesis as a pre-training task, and then align the
resulting rich scene priors with natural language via CLIP feature
distillation. We demonstrate the effectiveness of this approach by deploying
GeFF on a quadrupedal robot equipped with a manipulator. We evaluate GeFF's
ability to generalize to open-set objects as well as running time, when
performing open-vocabulary mobile manipulation in dynamic scenes.
65f11bae6750342ce0915664
Ri-Zhao Qiu
claimed_verified
2024-11-25T21:38:54.375Z
64eced8b6343a356327f5b45
/avatars/cf9de701a6fa893c7989773c4875c254.svg
Roger Qiu
user
RogerQi
65f11bae6750342ce0915665
Yafei Hu
claimed_verified
2024-03-13T13:26:27.653Z
63f3f1dca096536aeab25fcc
/avatars/7de71ca0c1b6711f4245fa75a0886225.svg
Yafei Hu
user
jeffreyhu
65f11bae6750342ce0915666
Ge Yang
admin_assigned
2024-03-13T13:59:01.984Z
5f03ee98ff69163f6f59e9e2
https://cdn-avatars.huggingface.co/v1/production/uploads/1594093169905-noauth.jpeg
Ge Yang
user
geyang
65f11bae6750342ce0915667
Yuchen Song
claimed_verified
2024-11-25T21:18:11.077Z
65482cb95f6c84d8f4c2267e
/avatars/c047bbb89db5bafe12f51e88a6f6dd0c.svg
Yuchen Song
user
YuchenEasonSong
65f11bae6750342ce0915668
Yang Fu
admin_assigned
2024-03-13T13:59:46.939Z
638d9a02a11654155ba9c7cb
/avatars/db85a28500554c1989b5f744bcd8b220.svg
Yang Fu
user
yangfu2
65f11bae6750342ce0915669
Jianglong Ye
admin_assigned
2024-03-13T13:59:53.438Z
634e2696d00bb5d92c3f4e86
https://cdn-avatars.huggingface.co/v1/production/uploads/1674216878324-634e2696d00bb5d92c3f4e86.jpeg
Jianglong Ye
user
jianglongye
65f11bae6750342ce091566a
Jiteng Mu
admin_assigned
2024-03-13T14:00:10.024Z
63a38a4a2262903957e12f14
/avatars/ad0e98736bdba5fa7a64c15c876c67a4.svg
JitengMu
user
JitengMu
65f11bae6750342ce091566b
Ruihan Yang
65f11bae6750342ce091566c
Nikolay Atanasov
65f11bae6750342ce091566d
Sebastian Scherer
65f11bae6750342ce091566e
Xiaolong Wang
Learning Generalizable Feature Fields for Mobile Manipulation
An open problem in mobile manipulation is how to represent objects and scenes
in a unified manner, so that robots can use it both for navigating in the
environment and manipulating objects. The latter requires capturing intricate
geometry while understanding fine-grained semantics, whereas the former
involves capturing the complexity inherit to an expansive physical scale. In
this work, we present GeFF (Generalizable Feature Fields), a scene-level
generalizable neural feature field that acts as a unified representation for
both navigation and manipulation that performs in real-time. To do so, we treat
generative novel view synthesis as a pre-training task, and then align the
resulting rich scene priors with natural language via CLIP feature
distillation. We demonstrate the effectiveness of this approach by deploying
GeFF on a quadrupedal robot equipped with a manipulator. We evaluate GeFF's
ability to generalize to open-set objects as well as running time, when
performing open-vocabulary mobile manipulation in dynamic scenes.
GeFF, a generalizable neural feature field, unifies navigation and manipulation tasks by leveraging generative novel view synthesis and CLIP feature distillation for real-time performance in dynamic scenes.
Generalizable Feature Fields neural feature field generative novel view synthesis CLIP feature distillation quadrupedal robot open-set objects open-vocabulary mobile manipulation