An Interactive Agent Foundation Model
The development of artificial intelligence systems is transitioning from
creating static, task-specific models to dynamic, agent-based systems capable
of performing well in a wide range of applications. We propose an Interactive
Agent Foundation Model that uses a novel multi-task agent training paradigm for
training AI agents across a wide range of domains, datasets, and tasks. Our
training paradigm unifies diverse pre-training strategies, including visual
masked auto-encoders, language modeling, and next-action prediction, enabling a
versatile and adaptable AI framework. We demonstrate the performance of our
framework across three separate domains -- Robotics, Gaming AI, and Healthcare.
Our model demonstrates its ability to generate meaningful and contextually
relevant outputs in each area. The strength of our approach lies in its
generality, leveraging a variety of data sources such as robotics sequences,
gameplay data, large-scale video datasets, and textual information for
effective multimodal and multi-task learning. Our approach provides a promising
avenue for developing generalist, action-taking, multimodal systems.
65c59916e286dbda4e966b55
Zane Durante
admin_assigned
2024-02-09T10:10:22.306Z
650259fda33ce3ae28620bf3
/avatars/5fb455b2e3a9523f07230d81d1947e15.svg
Zane Durante
user
ZaneDurante
65c59916e286dbda4e966b56
Bidipta Sarkar
claimed_verified
2024-02-09T08:05:04.783Z
63abbf74ad514ca8d14a0548
/avatars/b1357b73b8f9a8ff9908710ad64154ef.svg
Bidipta Sarkar
user
bidiptas
65c59916e286dbda4e966b57
Ran Gong
admin_assigned
2024-02-09T10:10:28.674Z
64acdaf75c6fc404f0d95e5b
/avatars/a4235fb6984e5945eb0e11c3ce581b59.svg
Ran Gong
user
nikepupu
65c59916e286dbda4e966b58
Rohan Taori
admin_assigned
2024-02-09T10:10:35.170Z
622fca0d8d46dde5d605f83e
https://cdn-avatars.huggingface.co/v1/production/uploads/1647489114181-622fca0d8d46dde5d605f83e.jpeg
Rohan Taori
user
rtaori
65c59916e286dbda4e966b59
Yusuke Noda
admin_assigned
2024-02-09T10:10:49.522Z
6430f78767688e5e12a408e4
/avatars/a7a10d30123495a3eb87d944ae9f2b55.svg
Yusuke Noda
user
yusukenoda
65c59916e286dbda4e966b5a
Paul Tang
65c59916e286dbda4e966b5b
Ehsan Adeli
claimed_verified
2024-02-12T08:39:55.429Z
65c673e773a47fa85dfc7ca3
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/4e1JO03Gh2l5s-8DW8MV2.jpeg
Ehsan
user
eadeli
65c59916e286dbda4e966b5c
Shrinidhi Kowshika Lakshmikanth
claimed_verified
2024-02-12T08:39:53.443Z
6456a0f04cfac49227918439
/avatars/9fa5191d1e9897f68ea2d21978b19cca.svg
Shrinidhi KL
user
klshrinidhi
65c59916e286dbda4e966b5d
Kevin Schulman
65c59916e286dbda4e966b5e
Arnold Milstein
65c59916e286dbda4e966b5f
Demetri Terzopoulos
65c59916e286dbda4e966b60
Ade Famoti
65c59916e286dbda4e966b61
Noboru Kuno
65c59916e286dbda4e966b62
Ashley Llorens
65c59916e286dbda4e966b63
Hoi Vo
admin_assigned
2024-02-09T10:16:06.984Z
631aae9bfac58c9c816636e1
/avatars/92d030f9eb552a334aa129e02d3f2ca6.svg
Hoi Vo
user
Hoivo63
65c59916e286dbda4e966b64
Katsu Ikeuchi
65c59916e286dbda4e966b65
Li Fei-Fei
65c59916e286dbda4e966b66
Jianfeng Gao
admin_assigned
2024-02-09T10:16:26.696Z
641904caf9d6f1d772ec7af7
/avatars/4a63eac71eb30f70b1a0e9d4708f26c1.svg
Jianfeng Gao
user
wyngjf
65c59916e286dbda4e966b67
Naoki Wake
admin_assigned
2024-02-09T10:16:33.608Z
650bd16424da30e773ec74ca
/avatars/f73536f9699d3759f8df8619c3084afc.svg
Wake
user
NaokiWake
65c59916e286dbda4e966b68
Qiuyuan Huang
An Interactive Agent Foundation Model
The development of artificial intelligence systems is transitioning from
creating static, task-specific models to dynamic, agent-based systems capable
of performing well in a wide range of applications. We propose an Interactive
Agent Foundation Model that uses a novel multi-task agent training paradigm for
training AI agents across a wide range of domains, datasets, and tasks. Our
training paradigm unifies diverse pre-training strategies, including visual
masked auto-encoders, language modeling, and next-action prediction, enabling a
versatile and adaptable AI framework. We demonstrate the performance of our
framework across three separate domains -- Robotics, Gaming AI, and Healthcare.
Our model demonstrates its ability to generate meaningful and contextually
relevant outputs in each area. The strength of our approach lies in its
generality, leveraging a variety of data sources such as robotics sequences,
gameplay data, large-scale video datasets, and textual information for
effective multimodal and multi-task learning. Our approach provides a promising
avenue for developing generalist, action-taking, multimodal systems.
The proposed Interactive Agent Foundation Model uses a multi-task agent training paradigm to develop versatile AI agents capable of performing well across diverse domains and tasks.
multi-task agent training paradigm visual masked auto-encoders language modeling next-action prediction multimodal learning multimodal systems