GNFactor: Multi-Task Real Robot Learning with Generalizable Neural
  Feature Fields
It is a long-standing problem in robotics to develop agents capable of
executing diverse manipulation tasks from visual observations in unstructured
real-world environments. To achieve this goal, the robot needs to have a
comprehensive understanding of the 3D structure and semantics of the scene. In
this work, we present GNFactor, a visual behavior cloning agent for
multi-task robotic manipulation with Generalizable Neural
feature Fields. GNFactor jointly optimizes a generalizable neural
field (GNF) as a reconstruction module and a Perceiver Transformer as a
decision-making module, leveraging a shared deep 3D voxel representation. To
incorporate semantics in 3D, the reconstruction module utilizes a
vision-language foundation model (e.g., Stable Diffusion) to distill
rich semantic information into the deep 3D voxel. We evaluate GNFactor on 3
real robot tasks and perform detailed ablations on 10 RLBench tasks with a
limited number of demonstrations. We observe a substantial improvement of
GNFactor over current state-of-the-art methods in seen and unseen tasks,
demonstrating the strong generalization ability of GNFactor. Our project
website is https://yanjieze.com/GNFactor/ .
64f1a063a5ca432503ccb3df
Yanjie Ze
claimed_verified
2023-09-01T09:49:10.373Z
63509bc859bfa9a85d4220aa
/avatars/ca2cc9b87f5ca5cd51606b2f9edf89d0.svg
Yanjie Ze
user
yjze
64f1a063a5ca432503ccb3e0
Ge Yan
64f1a063a5ca432503ccb3e1
Yueh-Hua Wu
64f1a063a5ca432503ccb3e2
Annabella Macaluso
64f1a063a5ca432503ccb3e3
Yuying Ge
64f1a063a5ca432503ccb3e4
Jianglong Ye
admin_assigned
2023-09-01T11:53:57.188Z
634e2696d00bb5d92c3f4e86
https://cdn-avatars.huggingface.co/v1/production/uploads/1674216878324-634e2696d00bb5d92c3f4e86.jpeg
Jianglong Ye
user
jianglongye
64f1a063a5ca432503ccb3e5
Nicklas Hansen
admin_assigned
2023-09-01T11:54:37.011Z
62d8ab94bf50eca8389bbc01
https://cdn-avatars.huggingface.co/v1/production/uploads/62d8ab94bf50eca8389bbc01/sXJupPzRO5ZV-BiQRdFeV.png
Nicklas Hansen
user
nicklashansen
64f1a063a5ca432503ccb3e6
Li Erran Li
64f1a063a5ca432503ccb3e7
Xiaolong Wang
GNFactor: Multi-Task Real Robot Learning with Generalizable Neural
  Feature Fields
It is a long-standing problem in robotics to develop agents capable of
executing diverse manipulation tasks from visual observations in unstructured
real-world environments. To achieve this goal, the robot needs to have a
comprehensive understanding of the 3D structure and semantics of the scene. In
this work, we present GNFactor, a visual behavior cloning agent for
multi-task robotic manipulation with Generalizable Neural
feature Fields. GNFactor jointly optimizes a generalizable neural
field (GNF) as a reconstruction module and a Perceiver Transformer as a
decision-making module, leveraging a shared deep 3D voxel representation. To
incorporate semantics in 3D, the reconstruction module utilizes a
vision-language foundation model (e.g., Stable Diffusion) to distill
rich semantic information into the deep 3D voxel. We evaluate GNFactor on 3
real robot tasks and perform detailed ablations on 10 RLBench tasks with a
limited number of demonstrations. We observe a substantial improvement of
GNFactor over current state-of-the-art methods in seen and unseen tasks,
demonstrating the strong generalization ability of GNFactor. Our project
website is https://yanjieze.com/GNFactor/ .
GNFactor, a visual behavior cloning agent for multi-task robotic manipulation, uses a generalizable neural field and a Perceiver Transformer to achieve high generalization on seen and unseen tasks.
generalizable neural field Perceiver Transformer deep 3D voxel representation vision-language foundation model Stable Diffusion RLBench tasks