SmartPlay : A Benchmark for LLMs as Intelligent Agents
Recent large language models (LLMs) have demonstrated great potential toward
intelligent agents and next-gen automation, but there currently lacks a
systematic benchmark for evaluating LLMs' abilities as agents. We introduce
SmartPlay: both a challenging benchmark and a methodology for evaluating LLMs
as agents. SmartPlay consists of 6 different games, including
Rock-Paper-Scissors, Tower of Hanoi, Minecraft. Each game features a unique
setting, providing up to 20 evaluation settings and infinite environment
variations. Each game in SmartPlay uniquely challenges a subset of 9 important
capabilities of an intelligent LLM agent, including reasoning with object
dependencies, planning ahead, spatial reasoning, learning from history, and
understanding randomness. The distinction between the set of capabilities each
game test allows us to analyze each capability separately. SmartPlay serves not
only as a rigorous testing ground for evaluating the overall performance of LLM
agents but also as a road-map for identifying gaps in current methodologies. We
release our benchmark at github.com/LLMsmartplay/SmartPlay
651cd98b9a47f703e6b2bee7
Yue Wu
651cd98b9a47f703e6b2bee8
Xuan Tang
651cd98b9a47f703e6b2bee9
Tom M. Mitchell
admin_assigned
2024-05-07T14:04:37.122Z
6439ac853ab54fdbab81c5c4
/avatars/5592fb0420e393a821ae60e6280e1553.svg
Tom Mitchell
user
tommmitchell
651cd98b9a47f703e6b2beea
Yuanzhi Li
admin_assigned
2024-05-07T14:05:47.561Z
641d5a313f04f9bf2da17e8c
/avatars/4035f73c69ddb045573ad96db4f13f04.svg
Yuanzhi Li
user
Uushizhu1234
SmartPlay : A Benchmark for LLMs as Intelligent Agents
Recent large language models (LLMs) have demonstrated great potential toward
intelligent agents and next-gen automation, but there currently lacks a
systematic benchmark for evaluating LLMs' abilities as agents. We introduce
SmartPlay: both a challenging benchmark and a methodology for evaluating LLMs
as agents. SmartPlay consists of 6 different games, including
Rock-Paper-Scissors, Tower of Hanoi, Minecraft. Each game features a unique
setting, providing up to 20 evaluation settings and infinite environment
variations. Each game in SmartPlay uniquely challenges a subset of 9 important
capabilities of an intelligent LLM agent, including reasoning with object
dependencies, planning ahead, spatial reasoning, learning from history, and
understanding randomness. The distinction between the set of capabilities each
game test allows us to analyze each capability separately. SmartPlay serves not
only as a rigorous testing ground for evaluating the overall performance of LLM
agents but also as a road-map for identifying gaps in current methodologies. We
release our benchmark at github.com/LLMsmartplay/SmartPlay
SmartPlay is a benchmark and evaluation methodology for large language models as agents, featuring 6 games that test various capabilities like reasoning, planning, spatial awareness, and learning.
large language models intelligent agents benchmark Rock-Paper-Scissors Tower of Hanoi Minecraft reasoning planning spatial reasoning learning from history understanding randomness