Octopus: Embodied Vision-Language Programmer from Environmental Feedback
Large vision-language models (VLMs) have achieved substantial progress in
multimodal perception and reasoning. Furthermore, when seamlessly integrated
into an embodied agent, it signifies a crucial stride towards the creation of
autonomous and context-aware systems capable of formulating plans and executing
commands with precision. In this paper, we introduce Octopus, a novel VLM
designed to proficiently decipher an agent's vision and textual task objectives
and to formulate intricate action sequences and generate executable code. Our
design allows the agent to adeptly handle a wide spectrum of tasks, ranging
from mundane daily chores in simulators to sophisticated interactions in
complex video games. Octopus is trained by leveraging GPT-4 to control an
explorative agent to generate training data, i.e., action blueprints and the
corresponding executable code, within our experimental environment called
OctoVerse. We also collect the feedback that allows the enhanced training
scheme of Reinforcement Learning with Environmental Feedback (RLEF). Through a
series of experiments, we illuminate Octopus's functionality and present
compelling results, and the proposed RLEF turns out to refine the agent's
decision-making. By open-sourcing our model architecture, simulator, and
dataset, we aspire to ignite further innovation and foster collaborative
applications within the broader embodied AI community.
6529471c019a20ed658f07d9
Jingkang Yang
claimed_verified
2023-10-13T14:42:54.592Z
62b5777f593a2c49da69dc02
https://cdn-avatars.huggingface.co/v1/production/uploads/1658152070753-62b5777f593a2c49da69dc02.jpeg
Jingkang Yang
user
Jingkang
6529471c019a20ed658f07da
Yuhao Dong
admin_assigned
2023-10-13T15:50:15.190Z
652965773a416e1f2173443b
https://cdn-avatars.huggingface.co/v1/production/uploads/652965773a416e1f2173443b/y9MB8YgHzbwCXAc4EI9T3.jpeg
Yuhao Dong
user
THUdyh
6529471c019a20ed658f07db
Shuai Liu
6529471c019a20ed658f07dc
Bo Li
claimed_verified
2024-08-09T07:49:39.633Z
62d3f7d84b0933c48f3cdd9c
https://cdn-avatars.huggingface.co/v1/production/uploads/62d3f7d84b0933c48f3cdd9c/rSoSyH0Td9fD9HVlyK7bh.jpeg
Bo Li
user
luodian
6529471c019a20ed658f07dd
Ziyue Wang
claimed_verified
2023-10-16T07:24:45.666Z
652961d24267f8c80220bbfb
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/0yCP-XYJu6PwTGkoyhKQB.jpeg
Ziyue Wang
user
annewzy
6529471c019a20ed658f07de
Chencheng Jiang
admin_assigned
2023-10-13T15:42:07.445Z
6470f5aad9360cd9d8e9310d
/avatars/86efd78d7ff12bded166f0a5e3fe2763.svg
Jiang Chencheng
user
jchencheng
6529471c019a20ed658f07df
Haoran Tan
claimed_verified
2023-10-21T09:22:51.468Z
652802a42949e6071c703550
https://cdn-avatars.huggingface.co/v1/production/uploads/652802a42949e6071c703550/dQF73cLlAoSLtgi8JzbnI.jpeg
Haoran Tan
user
Tan0118
6529471c019a20ed658f07e0
Jiamu Kang
6529471c019a20ed658f07e1
Yuanhan Zhang
admin_assigned
2023-10-13T15:39:27.007Z
62a993d80472c0b7f94027df
https://cdn-avatars.huggingface.co/v1/production/uploads/62a993d80472c0b7f94027df/j5vp-IwLA2YBexylUHiQU.png
Zhang Yuanhan
user
ZhangYuanhan
6529471c019a20ed658f07e2
Kaiyang Zhou
admin_assigned
2023-10-13T14:44:00.926Z
62ac6656de8bfbb93094b8fd
/avatars/1d8f445b6d5f9d43ddab81056bcb141e.svg
Kaiyang Zhou
user
kaiyangzhou
6529471c019a20ed658f07e3
Ziwei Liu
admin_assigned
2023-10-13T14:43:53.257Z
62ab1ac1d48b4d8b048a3473
https://cdn-avatars.huggingface.co/v1/production/uploads/1656826685333-62ab1ac1d48b4d8b048a3473.png
Ziwei Liu
user
liuziwei7
Octopus: Embodied Vision-Language Programmer from Environmental Feedback
Large vision-language models (VLMs) have achieved substantial progress in
multimodal perception and reasoning. Furthermore, when seamlessly integrated
into an embodied agent, it signifies a crucial stride towards the creation of
autonomous and context-aware systems capable of formulating plans and executing
commands with precision. In this paper, we introduce Octopus, a novel VLM
designed to proficiently decipher an agent's vision and textual task objectives
and to formulate intricate action sequences and generate executable code. Our
design allows the agent to adeptly handle a wide spectrum of tasks, ranging
from mundane daily chores in simulators to sophisticated interactions in
complex video games. Octopus is trained by leveraging GPT-4 to control an
explorative agent to generate training data, i.e., action blueprints and the
corresponding executable code, within our experimental environment called
OctoVerse. We also collect the feedback that allows the enhanced training
scheme of Reinforcement Learning with Environmental Feedback (RLEF). Through a
series of experiments, we illuminate Octopus's functionality and present
compelling results, and the proposed RLEF turns out to refine the agent's
decision-making. By open-sourcing our model architecture, simulator, and
dataset, we aspire to ignite further innovation and foster collaborative
applications within the broader embodied AI community.
Octopus, a novel vision-language model, integrates multimodal perception and reasoning to perform diverse tasks using GPT-4 and Reinforcement Learning with Environmental Feedback in an experimental environment called OctoVerse.
VLMs multimodal perception reasoning embodied agent action sequences executable code GPT-4 Reinforcement Learning with Environmental Feedback RLEF OctoVerse