Foundation Models in Robotics: Applications, Challenges, and the Future
We survey applications of pretrained foundation models in robotics.
Traditional deep learning models in robotics are trained on small datasets
tailored for specific tasks, which limits their adaptability across diverse
applications. In contrast, foundation models pretrained on internet-scale data
appear to have superior generalization capabilities, and in some instances
display an emergent ability to find zero-shot solutions to problems that are
not present in the training data. Foundation models may hold the potential to
enhance various components of the robot autonomy stack, from perception to
decision-making and control. For example, large language models can generate
code or provide common sense reasoning, while vision-language models enable
open-vocabulary visual recognition. However, significant open research
challenges remain, particularly around the scarcity of robot-relevant training
data, safety guarantees and uncertainty quantification, and real-time
execution. In this survey, we study recent papers that have used or built
foundation models to solve robotics problems. We explore how foundation models
contribute to improving robot capabilities in the domains of perception,
decision-making, and control. We discuss the challenges hindering the adoption
of foundation models in robot autonomy and provide opportunities and potential
pathways for future advancements. The GitHub project corresponding to this
paper (Preliminary release. We are committed to further enhancing and updating
this work to ensure its quality and relevance) can be found here:
https://github.com/robotics-survey/Awesome-Robotics-Foundation-Models
657a651ec222807b36e8641d
Roya Firoozi
657a651ec222807b36e8641e
Johnathan Tucker
657a651ec222807b36e8641f
Stephen Tian
admin_assigned
2023-12-14T13:37:58.119Z
64bf3b4a36eb058cd931a22e
/avatars/0946015f01b8774029dd694f558eab2c.svg
Stephen Tian
user
s-tian
657a651ec222807b36e86420
Anirudha Majumdar
657a651ec222807b36e86421
Jiankai Sun
admin_assigned
2023-12-14T13:38:10.641Z
63b5d1bb0d5913eee4869c0d
/avatars/ae397f54bbb3debc1f7903f4c6959ae6.svg
Jiankai Sun
user
zhenv5
657a651ec222807b36e86422
Weiyu Liu
657a651ec222807b36e86423
Yuke Zhu
657a651ec222807b36e86424
Shuran Song
657a651ec222807b36e86425
Ashish Kapoor
admin_assigned
2023-12-14T13:39:00.107Z
62a2eadd3621790e22a49b47
/avatars/b055657d60fffe310c05428ece2646e3.svg
Ashish Kapoor
user
ashishkapoor
657a651ec222807b36e86426
Karol Hausman
admin_assigned
2023-12-14T13:39:08.507Z
64c93e623a3f428da67d9310
/avatars/98f0b0f0c8ab9cd94a9f5650f197f6c3.svg
Karol Hausman
user
KarolHausman
657a651ec222807b36e86427
Brian Ichter
admin_assigned
2023-12-14T13:39:15.386Z
633a26ab474cfeb1a864dc56
/avatars/cd40accd894fc78810fc0d5108f413e9.svg
Brian I
user
brianichter
657a651ec222807b36e86428
Danny Driess
657a651ec222807b36e86429
Jiajun Wu
admin_assigned
2023-12-14T13:39:39.168Z
647b7b8345616c3de3627863
/avatars/8196364bffed71086ba9a0a4a62027d3.svg
Jiajun Wu
user
jiajunwu
657a651ec222807b36e8642a
Cewu Lu
657a651ec222807b36e8642b
Mac Schwager
Foundation Models in Robotics: Applications, Challenges, and the Future
We survey applications of pretrained foundation models in robotics.
Traditional deep learning models in robotics are trained on small datasets
tailored for specific tasks, which limits their adaptability across diverse
applications. In contrast, foundation models pretrained on internet-scale data
appear to have superior generalization capabilities, and in some instances
display an emergent ability to find zero-shot solutions to problems that are
not present in the training data. Foundation models may hold the potential to
enhance various components of the robot autonomy stack, from perception to
decision-making and control. For example, large language models can generate
code or provide common sense reasoning, while vision-language models enable
open-vocabulary visual recognition. However, significant open research
challenges remain, particularly around the scarcity of robot-relevant training
data, safety guarantees and uncertainty quantification, and real-time
execution. In this survey, we study recent papers that have used or built
foundation models to solve robotics problems. We explore how foundation models
contribute to improving robot capabilities in the domains of perception,
decision-making, and control. We discuss the challenges hindering the adoption
of foundation models in robot autonomy and provide opportunities and potential
pathways for future advancements. The GitHub project corresponding to this
paper (Preliminary release. We are committed to further enhancing and updating
this work to ensure its quality and relevance) can be found here:
https://github.com/robotics-survey/Awesome-Robotics-Foundation-Models
Pretrained foundation models, trained on internet-scale data, offer improved generalization and can enhance various aspects of robot autonomy, including perception, decision-making, and control, despite challenges in robotics-specific data, safety, and real-time execution.
pretrained foundation models internet-scale data generalization capabilities zero-shot solutions robot autonomy stack large language models common sense reasoning vision-language models open-vocabulary visual recognition robot-relevant training data safety guarantees uncertainty quantification real-time execution