Learn-by-interact: A Data-Centric Framework for Self-Adaptive Agents in
  Realistic Environments
Autonomous agents powered by large language models (LLMs) have the potential
to enhance human capabilities, assisting with digital tasks from sending emails
to performing data analysis. The abilities of existing LLMs at such tasks are
often hindered by the lack of high-quality agent data from the corresponding
environments they interact with. We propose Learn-by-interact, a data-centric
framework to adapt LLM agents to any given environments without human
annotations. Learn-by-interact synthesizes trajectories of agent-environment
interactions based on documentations, and constructs instructions by
summarizing or abstracting the interaction histories, a process called backward
construction. We assess the quality of our synthetic data by using them in both
training-based scenarios and training-free in-context learning (ICL), where we
craft innovative retrieval approaches optimized for agents. Extensive
experiments on SWE-bench, WebArena, OSWorld and Spider2-V spanning across
realistic coding, web, and desktop environments show the effectiveness of
Learn-by-interact in various downstream agentic tasks -- baseline results are
improved by up to 12.2\% for ICL with Claude-3.5 and 19.5\% for training with
Codestral-22B. We further demonstrate the critical role of backward
construction, which provides up to 14.0\% improvement for training. Our
ablation studies demonstrate the efficiency provided by our synthesized data in
ICL and the superiority of our retrieval pipeline over alternative approaches
like conventional retrieval-augmented generation (RAG). We expect that
Learn-by-interact will serve as a foundation for agent data synthesis as LLMs
are increasingly deployed at real-world environments.
67907dd5e1d8fc832b3e7b0f
Hongjin Su
admin_assigned
2025-01-22T15:51:36.858Z
632d53951538d4798a73c849
/avatars/b7d0a895e669bcd1303c4716b5401c36.svg
Hongjin SU
user
multi-train
67907dd5e1d8fc832b3e7b10
Ruoxi Sun
admin_assigned
2025-01-22T15:51:57.105Z
653093ad471261a41241e048
/avatars/54263ce75b102b81211d27b696920870.svg
Ruoxi Sun
user
brycesun
67907dd5e1d8fc832b3e7b11
Jinsung Yoon
admin_assigned
2025-01-22T15:52:16.664Z
64da5c05322a5774e085339a
/avatars/ef6a4325622799d474af778b92232555.svg
Jinsung Yoon
user
jsyoon0823
67907dd5e1d8fc832b3e7b12
Pengcheng Yin
admin_assigned
2025-01-22T15:52:34.099Z
65024921fa0dccbd859e55e9
/avatars/1f3c31c21ca692c2a63cf3437eac2507.svg
Pengcheng Yin
user
pchyin
67907dd5e1d8fc832b3e7b13
Tao Yu
67907dd5e1d8fc832b3e7b14
Sercan Ö. Arık
Learn-by-interact: A Data-Centric Framework for Self-Adaptive Agents in
  Realistic Environments
Autonomous agents powered by large language models (LLMs) have the potential
to enhance human capabilities, assisting with digital tasks from sending emails
to performing data analysis. The abilities of existing LLMs at such tasks are
often hindered by the lack of high-quality agent data from the corresponding
environments they interact with. We propose Learn-by-interact, a data-centric
framework to adapt LLM agents to any given environments without human
annotations. Learn-by-interact synthesizes trajectories of agent-environment
interactions based on documentations, and constructs instructions by
summarizing or abstracting the interaction histories, a process called backward
construction. We assess the quality of our synthetic data by using them in both
training-based scenarios and training-free in-context learning (ICL), where we
craft innovative retrieval approaches optimized for agents. Extensive
experiments on SWE-bench, WebArena, OSWorld and Spider2-V spanning across
realistic coding, web, and desktop environments show the effectiveness of
Learn-by-interact in various downstream agentic tasks -- baseline results are
improved by up to 12.2\% for ICL with Claude-3.5 and 19.5\% for training with
Codestral-22B. We further demonstrate the critical role of backward
construction, which provides up to 14.0\% improvement for training. Our
ablation studies demonstrate the efficiency provided by our synthesized data in
ICL and the superiority of our retrieval pipeline over alternative approaches
like conventional retrieval-augmented generation (RAG). We expect that
Learn-by-interact will serve as a foundation for agent data synthesis as LLMs
are increasingly deployed at real-world environments.
Learn-by-interact, a data-centric framework, synthesizes interaction data for LLM-based agents in various environments, improving performance in tasks like coding, web browsing, and desktop activity through backward construction and optimized retrieval methods.
LLMs Learn-by-interact agent data backward construction in-context learning SWE-bench WebArena OSWorld Spider2-V retrieval-augmented generation (RAG)