LEGENT: Open Platform for Embodied Agents
Despite advancements in Large Language Models (LLMs) and Large Multimodal
Models (LMMs), their integration into language-grounded, human-like embodied
agents remains incomplete, hindering complex real-life task performance in
physical environments. Existing integrations often feature limited open
sourcing, challenging collective progress in this field. We introduce LEGENT,
an open, scalable platform for developing embodied agents using LLMs and LMMs.
LEGENT offers a dual approach: a rich, interactive 3D environment with
communicable and actionable agents, paired with a user-friendly interface, and
a sophisticated data generation pipeline utilizing advanced algorithms to
exploit supervision from simulated worlds at scale. In our experiments, an
embryonic vision-language-action model trained on LEGENT-generated data
surpasses GPT-4V in embodied tasks, showcasing promising generalization
capabilities.
663058fb33ccd9e68c4b47d7
Zhili Cheng
admin_assigned
2024-04-30T07:38:32.998Z
635563ce525beaee688c4fc8
/avatars/f74eec56da76dd77a23649b76b857d8d.svg
Zhili Cheng
user
chengzl18
663058fb33ccd9e68c4b47d8
Zhitong Wang
admin_assigned
2024-04-30T07:38:44.960Z
64265a0ba5ec4a5cbc532cb1
/avatars/9c7fe642d510a3cb880b44ea8e129651.svg
Wang Zhitong
user
tommywang721
663058fb33ccd9e68c4b47d9
Jinyi Hu
claimed_verified
2024-04-30T07:27:38.900Z
637a06580a77f602dc4ac922
https://cdn-avatars.huggingface.co/v1/production/uploads/637a06580a77f602dc4ac922/JR2VHWhG1tj3TKKNzD8GD.jpeg
Jinyi Hu
user
JamesHujy
663058fb33ccd9e68c4b47da
Shengding Hu
admin_assigned
2024-04-30T07:38:51.615Z
6041bec6c15e823a685f2afc
/avatars/7d09fbe771d3c3312ac7eb6ef83ce053.svg
Shengding Hu
user
ShengdingHu
663058fb33ccd9e68c4b47db
An Liu
663058fb33ccd9e68c4b47dc
Yuge Tu
admin_assigned
2024-04-30T07:39:16.974Z
637f3220e0d5fd3199579458
https://cdn-avatars.huggingface.co/v1/production/uploads/637f3220e0d5fd3199579458/uaPvnmGqRcScz9RgoUcVY.png
Yuge Tu
user
tuyuge
663058fb33ccd9e68c4b47dd
Pengkai Li
663058fb33ccd9e68c4b47de
Lei Shi
663058fb33ccd9e68c4b47df
Zhiyuan Liu
claimed_verified
2025-06-02T07:59:13.458Z
6472f1f7485a7c8e1bcf507b
/avatars/79dca137583875191ea883b193b630b7.svg
Zhiyuan Liu
user
zibuyu9
663058fb33ccd9e68c4b47e0
Maosong Sun
LEGENT: Open Platform for Embodied Agents
Despite advancements in Large Language Models (LLMs) and Large Multimodal
Models (LMMs), their integration into language-grounded, human-like embodied
agents remains incomplete, hindering complex real-life task performance in
physical environments. Existing integrations often feature limited open
sourcing, challenging collective progress in this field. We introduce LEGENT,
an open, scalable platform for developing embodied agents using LLMs and LMMs.
LEGENT offers a dual approach: a rich, interactive 3D environment with
communicable and actionable agents, paired with a user-friendly interface, and
a sophisticated data generation pipeline utilizing advanced algorithms to
exploit supervision from simulated worlds at scale. In our experiments, an
embryonic vision-language-action model trained on LEGENT-generated data
surpasses GPT-4V in embodied tasks, showcasing promising generalization
capabilities.
LEGENT is an open platform for developing embodied agents using LLMs and LMMs, featuring a rich 3D environment and a data generation pipeline that outperforms GPT-4V in embodied tasks.
Large Language Models Large Multimodal Models embodied agents 3D environment data generation pipeline vision-language-action model GPT-4V