LangNav: Language as a Perceptual Representation for Navigation
We explore the use of language as a perceptual representation for
vision-and-language navigation. Our approach uses off-the-shelf vision systems
(for image captioning and object detection) to convert an agent's egocentric
panoramic view at each time step into natural language descriptions. We then
finetune a pretrained language model to select an action, based on the current
view and the trajectory history, that would best fulfill the navigation
instructions. In contrast to the standard setup which adapts a pretrained
language model to work directly with continuous visual features from pretrained
vision models, our approach instead uses (discrete) language as the perceptual
representation. We explore two use cases of our language-based navigation
(LangNav) approach on the R2R vision-and-language navigation benchmark:
generating synthetic trajectories from a prompted large language model (GPT-4)
with which to finetune a smaller language model; and sim-to-real transfer where
we transfer a policy learned on a simulated environment (ALFRED) to a
real-world environment (R2R). Our approach is found to improve upon strong
baselines that rely on visual features in settings where only a few gold
trajectories (10-100) are available, demonstrating the potential of using
language as a perceptual representation for navigation tasks.
652896a6aeb01f59a14f4b41
Bowen Pan
claimed_verified
2023-10-13T07:52:12.251Z
640e3a753830fd441c2c768d
https://cdn-avatars.huggingface.co/v1/production/uploads/640e3a753830fd441c2c768d/qztg6ML-c87VD8HajREsH.jpeg
Bowen Pan
user
bpan
652896a6aeb01f59a14f4b42
Rameswar Panda
extracted_pending
2023-10-13T01:00:22.575Z
648a7542ac1fbb8a1a654645
https://cdn-avatars.huggingface.co/v1/production/uploads/648a7542ac1fbb8a1a654645/jLPnY11EOuOa6ZHE7dVEX.jpeg
Rameswar Panda
user
rpand002
652896a6aeb01f59a14f4b43
SouYoung Jin
652896a6aeb01f59a14f4b44
Rogerio Feris
admin_assigned
2023-10-13T08:47:25.977Z
6426053ca5ec4a5cbc516567
/avatars/7b841e0c58aa746e662fc8c78e3c5537.svg
Rogerio Feris
user
rferis
652896a6aeb01f59a14f4b45
Aude Oliva
652896a6aeb01f59a14f4b46
Phillip Isola
admin_assigned
2023-10-13T08:47:40.566Z
623cb1f3ca87989bbf797eee
/avatars/13fbd5b123b8676561751f820317600c.svg
Phillip Isola
user
phillipi
652896a6aeb01f59a14f4b47
Yoon Kim
LangNav: Language as a Perceptual Representation for Navigation
We explore the use of language as a perceptual representation for
vision-and-language navigation. Our approach uses off-the-shelf vision systems
(for image captioning and object detection) to convert an agent's egocentric
panoramic view at each time step into natural language descriptions. We then
finetune a pretrained language model to select an action, based on the current
view and the trajectory history, that would best fulfill the navigation
instructions. In contrast to the standard setup which adapts a pretrained
language model to work directly with continuous visual features from pretrained
vision models, our approach instead uses (discrete) language as the perceptual
representation. We explore two use cases of our language-based navigation
(LangNav) approach on the R2R vision-and-language navigation benchmark:
generating synthetic trajectories from a prompted large language model (GPT-4)
with which to finetune a smaller language model; and sim-to-real transfer where
we transfer a policy learned on a simulated environment (ALFRED) to a
real-world environment (R2R). Our approach is found to improve upon strong
baselines that rely on visual features in settings where only a few gold
trajectories (10-100) are available, demonstrating the potential of using
language as a perceptual representation for navigation tasks.
Using natural language as a perceptual representation for vision-and-language navigation improves performance over visual feature-based approaches, especially with limited gold trajectories.
egocentric panoramic view image captioning object detection pretrained language model natural language descriptions vision-and-language navigation R2R benchmark GPT-4 sim-to-real transfer ALFRED environment