Agent Learning via Early Experience
A long-term goal of language agents is to learn and improve through their own
experience, ultimately outperforming humans in complex, real-world tasks.
However, training agents from experience data with reinforcement learning
remains difficult in many environments, which either lack verifiable rewards
(e.g., websites) or require inefficient long-horizon rollouts (e.g., multi-turn
tool use). As a result, most current agents rely on supervised fine-tuning on
expert data, which is challenging to scale and generalizes poorly. This
limitation stems from the nature of expert demonstrations: they capture only a
narrow range of scenarios and expose the agent to limited environment
diversity. We address this limitation with a middle-ground paradigm we call
early experience: interaction data generated by the agent's own actions, where
the resulting future states serve as supervision without reward signals. Within
this paradigm we study two strategies of using such data: (1) Implicit world
modeling, which uses collected states to ground the policy in environment
dynamics; and (2) Self-reflection, where the agent learns from its suboptimal
actions to improve reasoning and decision-making. We evaluate across eight
diverse environments and multiple model families. Our approaches consistently
improve effectiveness and out-of-domain generalization, highlighting the value
of early experience. Moreover, in environments with verifiable rewards, our
results provide promising signals that early experience offers a strong
foundation for subsequent reinforcement learning, positioning it as a practical
bridge between imitation learning and fully experience-driven agents.
68e86eaf95e8e6771df38925
Kai Zhang
claimed_verified
2025-10-10T04:32:42.421Z
63e0a50242591dda0b9dca5c
https://cdn-avatars.huggingface.co/v1/production/uploads/63e0a50242591dda0b9dca5c/c7cBPEBWQDFYimfGnO_SI.png
Kai Zhang
user
drogozhang
68e86eaf95e8e6771df38926
Xiangchao Chen
claimed_verified
2025-10-18T04:50:29.164Z
684c8b7349cb60c8c190d0a0
https://cdn-avatars.huggingface.co/v1/production/uploads/684c8b7349cb60c8c190d0a0/8916yYbA4GrAI065DWoow.jpeg
Xiangchao Chen
user
UlyssesXC
68e86eaf95e8e6771df38927
Bo Liu
claimed_verified
2025-10-10T04:32:47.903Z
635e3a76106f984574c36409
https://cdn-avatars.huggingface.co/v1/production/uploads/1667120725800-635e3a76106f984574c36409.png
Bo Liu
user
Benjamin-eecs
68e86eaf95e8e6771df38928
Tianci Xue
68e86eaf95e8e6771df38929
Zeyi Liao
68e86eaf95e8e6771df3892a
Zhihan Liu
68e86eaf95e8e6771df3892b
Xiyao Wang
claimed_verified
2025-10-10T04:32:45.647Z
655fed9fdef5905d38b84af3
/avatars/2cda4182dfd11a1e94743639e62328ea.svg
Xiyao Wang
user
russwang
68e86eaf95e8e6771df3892c
Yuting Ning
claimed_verified
2025-10-10T09:12:27.032Z
65ace92f64c9b93eca5c2bce
https://cdn-avatars.huggingface.co/v1/production/uploads/65ace92f64c9b93eca5c2bce/pG0JRXH-8zEy0IoaEnMNw.jpeg
Yuting Ning
user
nnnyt
68e86eaf95e8e6771df3892d
Zhaorun Chen
68e86eaf95e8e6771df3892e
Xiaohan Fu
claimed_verified
2025-10-13T10:10:04.897Z
661ad1d2e7b0ab12bcabf85c
/avatars/7220799b6a676c30b5374d01f1370406.svg
Xiaohan Fu
user
x5fu
68e86eaf95e8e6771df3892f
Jian Xie
68e86eaf95e8e6771df38930
Yuxuan Sun
68e86eaf95e8e6771df38931
Boyu Gou
claimed_verified
2025-10-10T04:32:38.644Z
6500870f1e14749e84f8f887
https://cdn-avatars.huggingface.co/v1/production/uploads/6500870f1e14749e84f8f887/wfvx4BZvh2OyW-vpq5jEy.jpeg
Boyu Gou
user
BoyuNLP
68e86eaf95e8e6771df38932
Qi Qi
68e86eaf95e8e6771df38933
Zihang Meng
68e86eaf95e8e6771df38934
Jianwei Yang
68e86eaf95e8e6771df38935
Ning Zhang
68e86eaf95e8e6771df38936
Xian Li
68e86eaf95e8e6771df38937
Ashish Shah
68e86eaf95e8e6771df38938
Dat Huynh
68e86eaf95e8e6771df38939
Hengduo Li
68e86eaf95e8e6771df3893a
Zi Yang
68e86eaf95e8e6771df3893b
Sara Cao
68e86eaf95e8e6771df3893c
Lawrence Jang
68e86eaf95e8e6771df3893d
Shuyan Zhou
68e86eaf95e8e6771df3893e
Jiacheng Zhu
claimed_verified
2025-10-13T10:10:02.691Z
653801b167325b6218ddfdc8
https://cdn-avatars.huggingface.co/v1/production/uploads/653801b167325b6218ddfdc8/y3DjEDpcvgnRC30201dFc.jpeg
Jiacheng Zhu
user
JiachengZhu
68e86eaf95e8e6771df3893f
Huan Sun
68e86eaf95e8e6771df38940
Jason Weston
68e86eaf95e8e6771df38941
Yu Su
claimed_verified
2025-10-11T13:48:10.987Z
6477a323dbc2a416f8b852b3
https://cdn-avatars.huggingface.co/v1/production/uploads/6477a323dbc2a416f8b852b3/mRKW5kT9GASORT4YnaZz0.jpeg
Yu Su
user
ysu-nlp
68e86eaf95e8e6771df38942
Yifan Wu
Agent Learning via Early Experience
A long-term goal of language agents is to learn and improve through their own
experience, ultimately outperforming humans in complex, real-world tasks.
However, training agents from experience data with reinforcement learning
remains difficult in many environments, which either lack verifiable rewards
(e.g., websites) or require inefficient long-horizon rollouts (e.g., multi-turn
tool use). As a result, most current agents rely on supervised fine-tuning on
expert data, which is challenging to scale and generalizes poorly. This
limitation stems from the nature of expert demonstrations: they capture only a
narrow range of scenarios and expose the agent to limited environment
diversity. We address this limitation with a middle-ground paradigm we call
early experience: interaction data generated by the agent's own actions, where
the resulting future states serve as supervision without reward signals. Within
this paradigm we study two strategies of using such data: (1) Implicit world
modeling, which uses collected states to ground the policy in environment
dynamics; and (2) Self-reflection, where the agent learns from its suboptimal
actions to improve reasoning and decision-making. We evaluate across eight
diverse environments and multiple model families. Our approaches consistently
improve effectiveness and out-of-domain generalization, highlighting the value
of early experience. Moreover, in environments with verifiable rewards, our
results provide promising signals that early experience offers a strong
foundation for subsequent reinforcement learning, positioning it as a practical
bridge between imitation learning and fully experience-driven agents.
Early experience, using agent-generated interaction data without reward signals, improves policy effectiveness and generalization, serving as a bridge between imitation learning and reinforcement learning.
reinforcement learning early experience implicit world modeling self-reflection out-of-domain generalization