NavGPT-2: Unleashing Navigational Reasoning Capability for Large
  Vision-Language Models
Capitalizing on the remarkable advancements in Large Language Models (LLMs),
there is a burgeoning initiative to harness LLMs for instruction following
robotic navigation. Such a trend underscores the potential of LLMs to
generalize navigational reasoning and diverse language understanding. However,
a significant discrepancy in agent performance is observed when integrating
LLMs in the Vision-and-Language navigation (VLN) tasks compared to previous
downstream specialist models. Furthermore, the inherent capacity of language to
interpret and facilitate communication in agent interactions is often
underutilized in these integrations. In this work, we strive to bridge the
divide between VLN-specialized models and LLM-based navigation paradigms, while
maintaining the interpretative prowess of LLMs in generating linguistic
navigational reasoning. By aligning visual content in a frozen LLM, we
encompass visual observation comprehension for LLMs and exploit a way to
incorporate LLMs and navigation policy networks for effective action
predictions and navigational reasoning. We demonstrate the data efficiency of
the proposed methods and eliminate the gap between LM-based agents and
state-of-the-art VLN specialists.
66999347319730566482c9c1
Gengze Zhou
claimed_verified
2024-07-22T07:10:35.326Z
6450bcd3673b2bcfaf8681af
/avatars/f5f93d780562d0772ec5dc1728945fcf.svg
Gengze Zhou
user
ZGZzz
66999347319730566482c9c2
Yicong Hong
66999347319730566482c9c3
Zun Wang
66999347319730566482c9c4
Xin Eric Wang
claimed_verified
2025-03-25T19:02:58.197Z
64679a226192d39142245e5e
/avatars/05abee0b6317f100923936ca2099e9eb.svg
Xin Eric Wang
user
xw-eric
66999347319730566482c9c5
Qi Wu
NavGPT-2: Unleashing Navigational Reasoning Capability for Large
  Vision-Language Models
Capitalizing on the remarkable advancements in Large Language Models (LLMs),
there is a burgeoning initiative to harness LLMs for instruction following
robotic navigation. Such a trend underscores the potential of LLMs to
generalize navigational reasoning and diverse language understanding. However,
a significant discrepancy in agent performance is observed when integrating
LLMs in the Vision-and-Language navigation (VLN) tasks compared to previous
downstream specialist models. Furthermore, the inherent capacity of language to
interpret and facilitate communication in agent interactions is often
underutilized in these integrations. In this work, we strive to bridge the
divide between VLN-specialized models and LLM-based navigation paradigms, while
maintaining the interpretative prowess of LLMs in generating linguistic
navigational reasoning. By aligning visual content in a frozen LLM, we
encompass visual observation comprehension for LLMs and exploit a way to
incorporate LLMs and navigation policy networks for effective action
predictions and navigational reasoning. We demonstrate the data efficiency of
the proposed methods and eliminate the gap between LM-based agents and
state-of-the-art VLN specialists.
The proposed method bridges the performance gap between Vision-and-Language (VLN) specialist models and Large Language Model (LLM)-based navigation by aligning visual content and integrating LLMs with policy networks for effective navigation.
Large Language Models LLMs instruction following robotic navigation Vision-and-Language navigation VLN tasks LLM-based navigation linguistic navigational reasoning action predictions navigational reasoning data efficiency