PIVOT: Iterative Visual Prompting Elicits Actionable Knowledge for VLMs
Vision language models (VLMs) have shown impressive capabilities across a
variety of tasks, from logical reasoning to visual understanding. This opens
the door to richer interaction with the world, for example robotic control.
However, VLMs produce only textual outputs, while robotic control and other
spatial tasks require outputting continuous coordinates, actions, or
trajectories. How can we enable VLMs to handle such settings without
fine-tuning on task-specific data?
  In this paper, we propose a novel visual prompting approach for VLMs that we
call Prompting with Iterative Visual Optimization (PIVOT), which casts tasks as
iterative visual question answering. In each iteration, the image is annotated
with a visual representation of proposals that the VLM can refer to (e.g.,
candidate robot actions, localizations, or trajectories). The VLM then selects
the best ones for the task. These proposals are iteratively refined, allowing
the VLM to eventually zero in on the best available answer. We investigate
PIVOT on real-world robotic navigation, real-world manipulation from images,
instruction following in simulation, and additional spatial inference tasks
such as localization. We find, perhaps surprisingly, that our approach enables
zero-shot control of robotic systems without any robot training data,
navigation in a variety of environments, and other capabilities. Although
current performance is far from perfect, our work highlights potentials and
limitations of this new regime and shows a promising approach for
Internet-Scale VLMs in robotic and spatial reasoning domains. Website:
pivot-prompt.github.io and HuggingFace:
https://huggingface.co/spaces/pivot-prompt/pivot-prompt-demo.
65cae6465495933ab081f5a5
Soroush Nasiriany
65cae6465495933ab081f5a6
Fei Xia
admin_assigned
2024-02-13T11:41:12.984Z
62cfa9ee9b96f22525c36d47
/avatars/8bfe3bf63f0609b3b6b7d7c5d7f6b3e9.svg
Fei Xia
user
fxia22
65cae6465495933ab081f5a7
Wenhao Yu
65cae6465495933ab081f5a8
Ted Xiao
65cae6465495933ab081f5a9
Jacky Liang
admin_assigned
2024-02-13T11:46:21.547Z
63017a70792a47211c3714c1
https://cdn-avatars.huggingface.co/v1/production/uploads/1667856283106-63017a70792a47211c3714c1.jpeg
Jacky Liang
user
jackyliang42
65cae6465495933ab081f5aa
Ishita Dasgupta
admin_assigned
2024-02-13T11:46:32.643Z
64f7f2871435cc7faa39104d
/avatars/8d42adabf6610eae5169be3ac42165e3.svg
Ishita Dasgupta
user
ishitadg
65cae6465495933ab081f5ab
Annie Xie
admin_assigned
2024-02-13T11:46:43.919Z
649e23bb3e5e75047640b3b5
/avatars/019d008a5bd7057b8665e1ef856f13a7.svg
Annie Xie
user
anniexie
65cae6465495933ab081f5ac
Danny Driess
65cae6465495933ab081f5ad
Ayzaan Wahid
admin_assigned
2024-02-13T11:46:57.735Z
63922fc073bf5173e3631bb7
/avatars/d672ca1cf5a541ec51f458b88eb9acec.svg
Ayzaan Wahid
user
ayzaan
65cae6465495933ab081f5ae
Zhuo Xu
65cae6465495933ab081f5af
Quan Vuong
admin_assigned
2024-02-13T11:50:57.619Z
63e1d6687fbb6ae4d4f4d980
/avatars/4959319e69d69e03a31ff4be326e602a.svg
Quan Vuong
user
quintonx
65cae6465495933ab081f5b0
Tingnan Zhang
65cae6465495933ab081f5b1
Tsang-Wei Edward Lee
65cae6465495933ab081f5b2
Kuang-Huei Lee
65cae6465495933ab081f5b3
Peng Xu
65cae6465495933ab081f5b4
Sean Kirmani
admin_assigned
2024-02-13T11:52:07.938Z
648c85f33ffc11989bddcce1
/avatars/bdde73b95a36e32cb2975656cca46022.svg
Sean Kirmani
user
skirmani
65cae6465495933ab081f5b5
Yuke Zhu
65cae6465495933ab081f5b6
Andy Zeng
admin_assigned
2024-02-13T11:52:34.206Z
62d5f9d705461e9459d20b91
/avatars/120917da2d8587674505163fb960e27d.svg
Andy Zeng
user
andyzeng
65cae6465495933ab081f5b7
Karol Hausman
admin_assigned
2024-02-13T11:52:40.789Z
64c93e623a3f428da67d9310
/avatars/98f0b0f0c8ab9cd94a9f5650f197f6c3.svg
Karol Hausman
user
KarolHausman
65cae6465495933ab081f5b8
Nicolas Heess
65cae6465495933ab081f5b9
Chelsea Finn
admin_assigned
2024-02-13T11:52:52.411Z
64ac22a9193f0a807deb673d
/avatars/fcac4912678ad3cb6e817d40bdee9aea.svg
Chelsea Finn
user
cbfinn
65cae6465495933ab081f5ba
Sergey Levine
65cae6465495933ab081f5bb
Brian Ichter
PIVOT: Iterative Visual Prompting Elicits Actionable Knowledge for VLMs
Vision language models (VLMs) have shown impressive capabilities across a
variety of tasks, from logical reasoning to visual understanding. This opens
the door to richer interaction with the world, for example robotic control.
However, VLMs produce only textual outputs, while robotic control and other
spatial tasks require outputting continuous coordinates, actions, or
trajectories. How can we enable VLMs to handle such settings without
fine-tuning on task-specific data?
  In this paper, we propose a novel visual prompting approach for VLMs that we
call Prompting with Iterative Visual Optimization (PIVOT), which casts tasks as
iterative visual question answering. In each iteration, the image is annotated
with a visual representation of proposals that the VLM can refer to (e.g.,
candidate robot actions, localizations, or trajectories). The VLM then selects
the best ones for the task. These proposals are iteratively refined, allowing
the VLM to eventually zero in on the best available answer. We investigate
PIVOT on real-world robotic navigation, real-world manipulation from images,
instruction following in simulation, and additional spatial inference tasks
such as localization. We find, perhaps surprisingly, that our approach enables
zero-shot control of robotic systems without any robot training data,
navigation in a variety of environments, and other capabilities. Although
current performance is far from perfect, our work highlights potentials and
limitations of this new regime and shows a promising approach for
Internet-Scale VLMs in robotic and spatial reasoning domains. Website:
pivot-prompt.github.io and HuggingFace:
https://huggingface.co/spaces/pivot-prompt/pivot-prompt-demo.
A method called PIVOT enables VLMs to perform iterative visual question answering to generate continuous outputs for tasks like robotic control, enabling zero-shot control without task-specific data.
visual prompting prompting with iterative visual optimization PIVOT visual question answering robotic navigation image-based manipulation instruction following spatial inference localization