Augmenting Autotelic Agents with Large Language Models
Humans learn to master open-ended repertoires of skills by imagining and
practicing their own goals. This autotelic learning process, literally the
pursuit of self-generated (auto) goals (telos), becomes more and more
open-ended as the goals become more diverse, abstract and creative. The
resulting exploration of the space of possible skills is supported by an
inter-individual exploration: goal representations are culturally evolved and
transmitted across individuals, in particular using language. Current
artificial agents mostly rely on predefined goal representations corresponding
to goal spaces that are either bounded (e.g. list of instructions), or
unbounded (e.g. the space of possible visual inputs) but are rarely endowed
with the ability to reshape their goal representations, to form new
abstractions or to imagine creative goals. In this paper, we introduce a
language model augmented autotelic agent (LMA3) that leverages a pretrained
language model (LM) to support the representation, generation and learning of
diverse, abstract, human-relevant goals. The LM is used as an imperfect model
of human cultural transmission; an attempt to capture aspects of humans'
common-sense, intuitive physics and overall interests. Specifically, it
supports three key components of the autotelic architecture: 1)~a relabeler
that describes the goals achieved in the agent's trajectories, 2)~a goal
generator that suggests new high-level goals along with their decomposition
into subgoals the agent already masters, and 3)~reward functions for each of
these goals. Without relying on any hand-coded goal representations, reward
functions or curriculum, we show that LMA3 agents learn to master a large
diversity of skills in a task-agnostic text-based environment.
646c25abdf2609a541c5067e
Cédric Colas
admin_assigned
2023-05-23T07:51:24.853Z
61dc8831819c5b746f66ad68
/avatars/25a2bc88ba74741f4579e9f41ad3cea2.svg
Cédric Colas
user
ccolas
646c25abdf2609a541c5067f
Laetitia Teodorescu
claimed_verified
2023-05-26T17:14:44.835Z
634310806f59b79da07b3d44
/avatars/7c229589badd39cc4f077cadf270bc74.svg
Laetitia
user
laetitia-teo
646c25abdf2609a541c50680
Pierre-Yves Oudeyer
admin_assigned
2023-05-23T07:52:09.465Z
60812cfb5acd42012ab4585f
/avatars/9ff76693e026f7a72312fc2b99729d8a.svg
Pierre-Yves Oudeyer
user
pyoudeyer
646c25abdf2609a541c50681
Xingdi Yuan
646c25abdf2609a541c50682
Marc-Alexandre Côté
admin_assigned
2023-05-23T07:53:03.345Z
62960afb91760de5e547e544
/avatars/bebed5926b04d066d4c0e49d246acd78.svg
Marc-Alexandre Côté
user
MarcCote
Augmenting Autotelic Agents with Large Language Models
Humans learn to master open-ended repertoires of skills by imagining and
practicing their own goals. This autotelic learning process, literally the
pursuit of self-generated (auto) goals (telos), becomes more and more
open-ended as the goals become more diverse, abstract and creative. The
resulting exploration of the space of possible skills is supported by an
inter-individual exploration: goal representations are culturally evolved and
transmitted across individuals, in particular using language. Current
artificial agents mostly rely on predefined goal representations corresponding
to goal spaces that are either bounded (e.g. list of instructions), or
unbounded (e.g. the space of possible visual inputs) but are rarely endowed
with the ability to reshape their goal representations, to form new
abstractions or to imagine creative goals. In this paper, we introduce a
language model augmented autotelic agent (LMA3) that leverages a pretrained
language model (LM) to support the representation, generation and learning of
diverse, abstract, human-relevant goals. The LM is used as an imperfect model
of human cultural transmission; an attempt to capture aspects of humans'
common-sense, intuitive physics and overall interests. Specifically, it
supports three key components of the autotelic architecture: 1)~a relabeler
that describes the goals achieved in the agent's trajectories, 2)~a goal
generator that suggests new high-level goals along with their decomposition
into subgoals the agent already masters, and 3)~reward functions for each of
these goals. Without relying on any hand-coded goal representations, reward
functions or curriculum, we show that LMA3 agents learn to master a large
diversity of skills in a task-agnostic text-based environment.
A language model augmented autotelic agent learns diverse, abstract goals through cultural transmission without predefined representations or curricula.
language model autotelic learning goal representations goal generator relabeler reward functions task-agnostic text-based environment