Generative World Explorer
Planning with partial observation is a central challenge in embodied AI. A
majority of prior works have tackled this challenge by developing agents that
physically explore their environment to update their beliefs about the world
state.In contrast, humans can imagine unseen parts of the world
through a mental exploration and revise their beliefs with imagined
observations. Such updated beliefs can allow them to make more informed
decisions, without necessitating the physical exploration of the world at all
times. To achieve this human-like ability, we introduce the Generative
World Explorer (Genex), an egocentric world exploration framework that allows
an agent to mentally explore a large-scale 3D world (e.g., urban scenes) and
acquire imagined observations to update its belief. This updated belief will
then help the agent to make a more informed decision at the current step. To
train Genex, we create a synthetic urban scene dataset, Genex-DB.
Our experimental results demonstrate that (1) Genex can generate
high-quality and consistent observations during long-horizon exploration of a
large virtual physical world and (2) the beliefs updated with the generated
observations can inform an existing decision-making model (e.g., an LLM agent)
to make better plans.
673bfe4c2bf4163ef13f497f
Taiming Lu
claimed_verified
2024-11-19T08:39:37.063Z
656a9b9f9496f21be8271f1b
https://cdn-avatars.huggingface.co/v1/production/uploads/656a9b9f9496f21be8271f1b/e7p5kx2YNs_A1AJYSOlio.png
TaiMing
user
TaiMingLu
673bfe4c2bf4163ef13f4980
Tianmin Shu
admin_assigned
2024-11-19T13:07:10.536Z
618af79af281834c95311402
/avatars/58ba086eb2309657d823b5768616eafe.svg
Tianmin Shu
user
tshu
673bfe4c2bf4163ef13f4981
Alan Yuille
673bfe4c2bf4163ef13f4982
Daniel Khashabi
admin_assigned
2024-11-19T13:07:23.603Z
5f6540c65e78cc6b0ed3199d
/avatars/0280d4df417855965a0964d22766c012.svg
Daniel Khashabi
user
danyaljj
673bfe4c2bf4163ef13f4983
Jieneng Chen
claimed_verified
2024-11-19T08:39:33.842Z
660c9ac4b202fcf3892f62fa
/avatars/7314fd5f3f642096d0e37d3194f1aa7e.svg
Jieneng Chen
user
jienengchen
Generative World Explorer
Planning with partial observation is a central challenge in embodied AI. A
majority of prior works have tackled this challenge by developing agents that
physically explore their environment to update their beliefs about the world
state.In contrast, humans can imagine unseen parts of the world
through a mental exploration and revise their beliefs with imagined
observations. Such updated beliefs can allow them to make more informed
decisions, without necessitating the physical exploration of the world at all
times. To achieve this human-like ability, we introduce the Generative
World Explorer (Genex), an egocentric world exploration framework that allows
an agent to mentally explore a large-scale 3D world (e.g., urban scenes) and
acquire imagined observations to update its belief. This updated belief will
then help the agent to make a more informed decision at the current step. To
train Genex, we create a synthetic urban scene dataset, Genex-DB.
Our experimental results demonstrate that (1) Genex can generate
high-quality and consistent observations during long-horizon exploration of a
large virtual physical world and (2) the beliefs updated with the generated
observations can inform an existing decision-making model (e.g., an LLM agent)
to make better plans.
Generative World Explorer (Genex) enables agents to mentally explore 3D environments using imagined observations, updating their beliefs without physical exploration to improve decision-making.
Generative World Explorer Genex egocentric world exploration belief updating imagined observations large-scale 3D world synthesis Genex-DB long-horizon exploration decision-making model LLM agent