Collaborative Instance Navigation: Leveraging Agent Self-Dialogue to
  Minimize User Input
Existing embodied instance goal navigation tasks, driven by natural language,
assume human users to provide complete and nuanced instance descriptions prior
to the navigation, which can be impractical in the real world as human
instructions might be brief and ambiguous. To bridge this gap, we propose a new
task, Collaborative Instance Navigation (CoIN), with dynamic agent-human
interaction during navigation to actively resolve uncertainties about the
target instance in natural, template-free, open-ended dialogues. To address
CoIN, we propose a novel method, Agent-user Interaction with UncerTainty
Awareness (AIUTA), leveraging the perception capability of Vision Language
Models (VLMs) and the capability of Large Language Models (LLMs). First, upon
object detection, a Self-Questioner model initiates a self-dialogue to obtain a
complete and accurate observation description, while a novel uncertainty
estimation technique mitigates inaccurate VLM perception. Then, an Interaction
Trigger module determines whether to ask a question to the user, continue or
halt navigation, minimizing user input. For evaluation, we introduce
CoIN-Bench, a benchmark supporting both real and simulated humans. AIUTA
achieves competitive performance in instance navigation against
state-of-the-art methods, demonstrating great flexibility in handling user
inputs.
674ec96f8b4bb24d65e3e782
Francesco Taioli
extracted_confirmed
2024-12-03T09:05:34.387Z
63ab07a60ed3c32528492782
https://cdn-avatars.huggingface.co/v1/production/uploads/63ab07a60ed3c32528492782/nwNxQWdnDzFkpc48fYH0A.jpeg
Francesco Taioli
user
ftaioli
674ec96f8b4bb24d65e3e783
Edoardo Zorzi
claimed_verified
2024-12-03T12:00:11.637Z
6621462611c923d051d62072
https://cdn-avatars.huggingface.co/v1/production/uploads/6621462611c923d051d62072/7oD7SSCY3MSQeTsvcM1IK.png
Edoardo
user
e-zorzi
674ec96f8b4bb24d65e3e784
Gianni Franchi
674ec96f8b4bb24d65e3e785
Alberto Castellini
674ec96f8b4bb24d65e3e786
Alessandro Farinelli
674ec96f8b4bb24d65e3e787
Marco Cristani
674ec96f8b4bb24d65e3e788
Yiming Wang
extracted_confirmed
2025-03-18T14:41:27.857Z
67d98541faa3f91a1a7491eb
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/nzdKQB350VBFZE0QXNLuj.png
yiming wang
user
yiming-wang
Collaborative Instance Navigation: Leveraging Agent Self-Dialogue to
  Minimize User Input
Existing embodied instance goal navigation tasks, driven by natural language,
assume human users to provide complete and nuanced instance descriptions prior
to the navigation, which can be impractical in the real world as human
instructions might be brief and ambiguous. To bridge this gap, we propose a new
task, Collaborative Instance Navigation (CoIN), with dynamic agent-human
interaction during navigation to actively resolve uncertainties about the
target instance in natural, template-free, open-ended dialogues. To address
CoIN, we propose a novel method, Agent-user Interaction with UncerTainty
Awareness (AIUTA), leveraging the perception capability of Vision Language
Models (VLMs) and the capability of Large Language Models (LLMs). First, upon
object detection, a Self-Questioner model initiates a self-dialogue to obtain a
complete and accurate observation description, while a novel uncertainty
estimation technique mitigates inaccurate VLM perception. Then, an Interaction
Trigger module determines whether to ask a question to the user, continue or
halt navigation, minimizing user input. For evaluation, we introduce
CoIN-Bench, a benchmark supporting both real and simulated humans. AIUTA
achieves competitive performance in instance navigation against
state-of-the-art methods, demonstrating great flexibility in handling user
inputs.
New Collaborative Instance Navigation (CoIN) task utilizes dynamic agent-human dialogue and AIUTA method combining VLMs and LLMs to handle uncertain and ambiguous human instructions effectively.
Collaborative Instance Navigation CoIN Agent-user Interaction with UncerTainty Awareness AIUTA Vision Language Models VLMs Large Language Models LLMs Self-Questioner self-dialogue uncertainty estimation Interaction Trigger module CoIN-Bench instance navigation