RoboVQA: Multimodal Long-Horizon Reasoning for Robotics
We present a scalable, bottom-up and intrinsically diverse data collection
scheme that can be used for high-level reasoning with long and medium horizons
and that has 2.2x higher throughput compared to traditional narrow top-down
step-by-step collection. We collect realistic data by performing any user
requests within the entirety of 3 office buildings and using multiple robot and
human embodiments. With this data, we show that models trained on all
embodiments perform better than ones trained on the robot data only, even when
evaluated solely on robot episodes. We find that for a fixed collection budget
it is beneficial to take advantage of cheaper human collection along with robot
collection. We release a large and highly diverse (29,520 unique instructions)
dataset dubbed RoboVQA containing 829,502 (video, text) pairs for
robotics-focused visual question answering. We also demonstrate how evaluating
real robot experiments with an intervention mechanism enables performing tasks
to completion, making it deployable with human oversight even if imperfect
while also providing a single performance metric. We demonstrate a single
video-conditioned model named RoboVQA-VideoCoCa trained on our dataset that is
capable of performing a variety of grounded high-level reasoning tasks in broad
realistic settings with a cognitive intervention rate 46% lower than the
zero-shot state of the art visual language model (VLM) baseline and is able to
guide real robots through long-horizon tasks. The performance gap with
zero-shot state-of-the-art models indicates that a lot of grounded data remains
to be collected for real-world deployment, emphasizing the critical need for
scalable data collection approaches. Finally, we show that video VLMs
significantly outperform single-image VLMs with an average error rate reduction
of 19% across all VQA tasks. Data and videos available at
https://robovqa.github.io
6544fab22a13610acc1d376f
Pierre Sermanet
6544fab22a13610acc1d3770
Tianli Ding
6544fab22a13610acc1d3771
Jeffrey Zhao
6544fab22a13610acc1d3772
Fei Xia
claimed_verified
2024-01-23T08:09:49.294Z
62cfa9ee9b96f22525c36d47
/avatars/8bfe3bf63f0609b3b6b7d7c5d7f6b3e9.svg
Fei Xia
user
fxia22
6544fab22a13610acc1d3773
Debidatta Dwibedi
6544fab22a13610acc1d3774
Keerthana Gopalakrishnan
admin_assigned
2023-11-03T16:10:35.935Z
632f7dee34c562cb1d380fdc
/avatars/d689e260535b8aa775654160028d4ec8.svg
Keerthana Gopalakrishnan
user
keerthanpg
6544fab22a13610acc1d3775
Christine Chan
6544fab22a13610acc1d3776
Gabriel Dulac-Arnold
6544fab22a13610acc1d3777
Sharath Maddineni
6544fab22a13610acc1d3778
Nikhil J Joshi
admin_assigned
2023-11-03T16:11:03.603Z
63c15485894342c8964c481d
/avatars/cd938a12eb73a8395872f4058d85b0c5.svg
nikhil joshi
user
valeska
6544fab22a13610acc1d3779
Pete Florence
6544fab22a13610acc1d377a
Wei Han
6544fab22a13610acc1d377b
Robert Baruch
6544fab22a13610acc1d377c
Yao Lu
6544fab22a13610acc1d377d
Suvir Mirchandani
6544fab22a13610acc1d377e
Peng Xu
6544fab22a13610acc1d377f
Pannag Sanketi
claimed_verified
2024-08-09T07:49:31.531Z
66b56f73fc66a466e6104bfb
/avatars/74855aaf580144fafead4b93d6a24a46.svg
Pannag Sanketi
user
prsanky
6544fab22a13610acc1d3780
Karol Hausman
admin_assigned
2023-11-03T16:11:36.589Z
64c93e623a3f428da67d9310
/avatars/98f0b0f0c8ab9cd94a9f5650f197f6c3.svg
Karol Hausman
user
KarolHausman
6544fab22a13610acc1d3781
Izhak Shafran
6544fab22a13610acc1d3782
Brian Ichter
admin_assigned
2023-11-03T16:11:47.187Z
633a26ab474cfeb1a864dc56
/avatars/cd40accd894fc78810fc0d5108f413e9.svg
Brian I
user
brianichter
6544fab22a13610acc1d3783
Yuan Cao
admin_assigned
2023-11-03T16:12:30.777Z
6433a8c37b8247480105cf7f
/avatars/a19536f3f3b8eae3609648bea112b381.svg
Yuan Cao
user
caoyuan33
RoboVQA: Multimodal Long-Horizon Reasoning for Robotics
We present a scalable, bottom-up and intrinsically diverse data collection
scheme that can be used for high-level reasoning with long and medium horizons
and that has 2.2x higher throughput compared to traditional narrow top-down
step-by-step collection. We collect realistic data by performing any user
requests within the entirety of 3 office buildings and using multiple robot and
human embodiments. With this data, we show that models trained on all
embodiments perform better than ones trained on the robot data only, even when
evaluated solely on robot episodes. We find that for a fixed collection budget
it is beneficial to take advantage of cheaper human collection along with robot
collection. We release a large and highly diverse (29,520 unique instructions)
dataset dubbed RoboVQA containing 829,502 (video, text) pairs for
robotics-focused visual question answering. We also demonstrate how evaluating
real robot experiments with an intervention mechanism enables performing tasks
to completion, making it deployable with human oversight even if imperfect
while also providing a single performance metric. We demonstrate a single
video-conditioned model named RoboVQA-VideoCoCa trained on our dataset that is
capable of performing a variety of grounded high-level reasoning tasks in broad
realistic settings with a cognitive intervention rate 46% lower than the
zero-shot state of the art visual language model (VLM) baseline and is able to
guide real robots through long-horizon tasks. The performance gap with
zero-shot state-of-the-art models indicates that a lot of grounded data remains
to be collected for real-world deployment, emphasizing the critical need for
scalable data collection approaches. Finally, we show that video VLMs
significantly outperform single-image VLMs with an average error rate reduction
of 19% across all VQA tasks. Data and videos available at
https://robovqa.github.io
A scalable data collection scheme using both robots and humans within office buildings enhances the performance of models in robotics-focused visual question answering, particularly for long-horizon tasks, and demonstrates the importance of grounded data for real-world deployment.
data collection high-level reasoning long and medium horizons robot and human embodiments robotics-focused visual question answering RoboVQA dataset video-conditioned model RoboVQA-VideoCoCa cognitive intervention grounded high-level reasoning video VLMs single-image VLMs