Artificial Generational Intelligence: Cultural Accumulation in
  Reinforcement Learning
Cultural accumulation drives the open-ended and diverse progress in
capabilities spanning human history. It builds an expanding body of knowledge
and skills by combining individual exploration with inter-generational
information transmission. Despite its widespread success among humans, the
capacity for artificial learning agents to accumulate culture remains
under-explored. In particular, approaches to reinforcement learning typically
strive for improvements over only a single lifetime. Generational algorithms
that do exist fail to capture the open-ended, emergent nature of cultural
accumulation, which allows individuals to trade-off innovation and imitation.
Building on the previously demonstrated ability for reinforcement learning
agents to perform social learning, we find that training setups which balance
this with independent learning give rise to cultural accumulation. These
accumulating agents outperform those trained for a single lifetime with the
same cumulative experience. We explore this accumulation by constructing two
models under two distinct notions of a generation: episodic generations, in
which accumulation occurs via in-context learning and train-time generations,
in which accumulation occurs via in-weights learning. In-context and in-weights
cultural accumulation can be interpreted as analogous to knowledge and skill
accumulation, respectively. To the best of our knowledge, this work is the
first to present general models that achieve emergent cultural accumulation in
reinforcement learning, opening up new avenues towards more open-ended learning
systems, as well as presenting new opportunities for modelling human culture.
665e92b6a62bca96e5a32905
Jonathan Cook
admin_assigned
2024-06-04T08:35:47.465Z
64c900bbcbcae9f106e0eb53
/avatars/4028efd3476980ae54f8bb707db96ec0.svg
Jonathan Cook
user
jcook2
665e92b6a62bca96e5a32906
Chris Lu
claimed_verified
2024-06-04T19:28:45.965Z
64cbfc1d3576a06fa252736d
/avatars/7345beafa590d5c9d5ac80cb33a9f0ab.svg
Chris Lu
user
chrlu
665e92b6a62bca96e5a32907
Edward Hughes
admin_assigned
2024-06-04T08:36:10.715Z
64dcad22249785efad8da97b
/avatars/11ebe5dec1e5d2e3c7cd00ec684e2319.svg
Edward Hughes
user
visible-ed
665e92b6a62bca96e5a32908
Joel Z. Leibo
admin_assigned
2024-06-04T08:36:20.150Z
66414cb58732bcd0762d41cf
/avatars/0f8489364711906be6c60b850e159c03.svg
Joel Z Leibo
user
jzleibo
665e92b6a62bca96e5a32909
Jakob Foerster
Artificial Generational Intelligence: Cultural Accumulation in
  Reinforcement Learning
Cultural accumulation drives the open-ended and diverse progress in
capabilities spanning human history. It builds an expanding body of knowledge
and skills by combining individual exploration with inter-generational
information transmission. Despite its widespread success among humans, the
capacity for artificial learning agents to accumulate culture remains
under-explored. In particular, approaches to reinforcement learning typically
strive for improvements over only a single lifetime. Generational algorithms
that do exist fail to capture the open-ended, emergent nature of cultural
accumulation, which allows individuals to trade-off innovation and imitation.
Building on the previously demonstrated ability for reinforcement learning
agents to perform social learning, we find that training setups which balance
this with independent learning give rise to cultural accumulation. These
accumulating agents outperform those trained for a single lifetime with the
same cumulative experience. We explore this accumulation by constructing two
models under two distinct notions of a generation: episodic generations, in
which accumulation occurs via in-context learning and train-time generations,
in which accumulation occurs via in-weights learning. In-context and in-weights
cultural accumulation can be interpreted as analogous to knowledge and skill
accumulation, respectively. To the best of our knowledge, this work is the
first to present general models that achieve emergent cultural accumulation in
reinforcement learning, opening up new avenues towards more open-ended learning
systems, as well as presenting new opportunities for modelling human culture.
Reinforcement learning agents can accumulate cultural knowledge and skills through social learning and generational training, outperforming single-lifetime training.
cultural accumulation reinforcement learning generational algorithms social learning in-context learning in-weights learning emergent cultural accumulation open-ended learning systems human culture