Ghost in the Minecraft: Generally Capable Agents for Open-World
  Enviroments via Large Language Models with Text-based Knowledge and Memory
The captivating realm of Minecraft has attracted substantial research
interest in recent years, serving as a rich platform for developing intelligent
agents capable of functioning in open-world environments. However, the current
research landscape predominantly focuses on specific objectives, such as the
popular "ObtainDiamond" task, and has not yet shown effective generalization to
a broader spectrum of tasks. Furthermore, the current leading success rate for
the "ObtainDiamond" task stands at around 20%, highlighting the limitations of
Reinforcement Learning (RL) based controllers used in existing methods. To
tackle these challenges, we introduce Ghost in the Minecraft (GITM), a novel
framework integrates Large Language Models (LLMs) with text-based knowledge and
memory, aiming to create Generally Capable Agents (GCAs) in Minecraft. These
agents, equipped with the logic and common sense capabilities of LLMs, can
skillfully navigate complex, sparse-reward environments with text-based
interactions. We develop a set of structured actions and leverage LLMs to
generate action plans for the agents to execute. The resulting LLM-based agent
markedly surpasses previous methods, achieving a remarkable improvement of
+47.5% in success rate on the "ObtainDiamond" task, demonstrating superior
robustness compared to traditional RL-based controllers. Notably, our agent is
the first to procure all items in the Minecraft Overworld technology tree,
demonstrating its extensive capabilities. GITM does not need any GPU for
training, but a single CPU node with 32 CPU cores is enough. This research
shows the potential of LLMs in developing capable agents for handling
long-horizon, complex tasks and adapting to uncertainties in open-world
environments. See the project website at https://github.com/OpenGVLab/GITM.
64758bcca855203d8fefcde4
Xizhou Zhu
64758bcca855203d8fefcde5
Yuntao Chen
64758bcca855203d8fefcde6
Hao Tian
64758bcca855203d8fefcde7
Chenxin Tao
extracted_confirmed
2023-05-30T06:32:52.799Z
632dab84fdb35759ea6646a0
/avatars/857b0b4d115aa5ab2f143e60b0e4edc6.svg
Yuntao Chen
user
YuntaoChen
64758bcca855203d8fefcde8
Weijie Su
claimed_verified
2023-05-30T07:46:41.610Z
63e4562f9db5da2dc1f3b520
/avatars/f4eecf1396b05e1c72436e7026d85cef.svg
Weijie Su
user
jackroos
64758bcca855203d8fefcde9
Chenyu Yang
claimed_verified
2025-05-30T07:01:02.975Z
637f347a52229c639211bee8
https://cdn-avatars.huggingface.co/v1/production/uploads/637f347a52229c639211bee8/I9_PET-_6SJQJ6hXrACV4.jpeg
Chenyu Yang
user
cyyang822
64758bcca855203d8fefcdea
Gao Huang
64758bcca855203d8fefcdeb
Bin Li
64758bcca855203d8fefcdec
Lewei Lu
64758bcca855203d8fefcded
Xiaogang Wang
64758bcca855203d8fefcdee
Yu Qiao
64758bcca855203d8fefcdef
Zhaoxiang Zhang
64758bcca855203d8fefcdf0
Jifeng Dai
claimed_verified
2023-05-30T07:46:44.632Z
64686f7172d9180d4ac8b4e4
/avatars/db67dd6c4b2b41054ddcce5a18ade6f8.svg
Jifeng Dai
user
daijifeng
Ghost in the Minecraft: Generally Capable Agents for Open-World
  Enviroments via Large Language Models with Text-based Knowledge and Memory
The captivating realm of Minecraft has attracted substantial research
interest in recent years, serving as a rich platform for developing intelligent
agents capable of functioning in open-world environments. However, the current
research landscape predominantly focuses on specific objectives, such as the
popular "ObtainDiamond" task, and has not yet shown effective generalization to
a broader spectrum of tasks. Furthermore, the current leading success rate for
the "ObtainDiamond" task stands at around 20%, highlighting the limitations of
Reinforcement Learning (RL) based controllers used in existing methods. To
tackle these challenges, we introduce Ghost in the Minecraft (GITM), a novel
framework integrates Large Language Models (LLMs) with text-based knowledge and
memory, aiming to create Generally Capable Agents (GCAs) in Minecraft. These
agents, equipped with the logic and common sense capabilities of LLMs, can
skillfully navigate complex, sparse-reward environments with text-based
interactions. We develop a set of structured actions and leverage LLMs to
generate action plans for the agents to execute. The resulting LLM-based agent
markedly surpasses previous methods, achieving a remarkable improvement of
+47.5% in success rate on the "ObtainDiamond" task, demonstrating superior
robustness compared to traditional RL-based controllers. Notably, our agent is
the first to procure all items in the Minecraft Overworld technology tree,
demonstrating its extensive capabilities. GITM does not need any GPU for
training, but a single CPU node with 32 CPU cores is enough. This research
shows the potential of LLMs in developing capable agents for handling
long-horizon, complex tasks and adapting to uncertainties in open-world
environments. See the project website at https://github.com/OpenGVLab/GITM.
A framework integrating Large Language Models with text-based knowledge achieves superior performance in Minecraft by surpassing traditional RL methods and completing complex tasks.
Large Language Models text-based knowledge memory Generally Capable Agents action plans long-horizon tasks