Scaling LLM Multi-turn RL with End-to-end Summarization-based Context
  Management
We study reinforcement learning (RL) fine-tuning of large language model
(LLM) agents for long-horizon multi-turn tool use, where context length quickly
becomes a fundamental bottleneck. Existing RL pipelines can suffer from
degraded instruction following, excessive rollout costs, and most importantly,
strict context limits. To address these challenges, we introduce
summarization-based context management to training. In specific, it
periodically compresses the tool using history by LLM-generated summaries that
retain task-relevant information to keep a compact context while enabling the
agent to scale beyond the fixed context window. Building on this formulation,
we derive a policy gradient representation that seamlessly enables standard LLM
RL infrastructures to optimize both tool-use behaviors as well as summarization
strategies in an end-to-end fashion. We instantiate this framework with
SUmmarization augmented Policy Optimization
(SUPO), an LLM RL algorithm that enables long-horizon training beyond
a fixed context limit. Experiments on interactive function calling and
searching tasks demonstrate that SUPO significantly improves the
success rate while maintaining the same or even lower working context length
compared to baselines. We also demonstrate that for complex searching tasks,
SUPO can further improve the evaluation performance when scaling
test-time maximum round of summarization beyond that of training time. Our
results establish summarization-based context management as a principled and
scalable approach for training RL agents beyond a fixed context length limit.
68f0412436f8b025381e18a7
Miao Lu
68f0412436f8b025381e18a8
Weiwei Sun
68f0412436f8b025381e18a9
Weihua Du
68f0412436f8b025381e18aa
Zhan Ling
68f0412436f8b025381e18ab
Xuesong Yao
68f0412436f8b025381e18ac
Kang Liu
68f0412436f8b025381e18ad
Jiecao Chen
Scaling LLM Multi-turn RL with End-to-end Summarization-based Context
  Management
We study reinforcement learning (RL) fine-tuning of large language model
(LLM) agents for long-horizon multi-turn tool use, where context length quickly
becomes a fundamental bottleneck. Existing RL pipelines can suffer from
degraded instruction following, excessive rollout costs, and most importantly,
strict context limits. To address these challenges, we introduce
summarization-based context management to training. In specific, it
periodically compresses the tool using history by LLM-generated summaries that
retain task-relevant information to keep a compact context while enabling the
agent to scale beyond the fixed context window. Building on this formulation,
we derive a policy gradient representation that seamlessly enables standard LLM
RL infrastructures to optimize both tool-use behaviors as well as summarization
strategies in an end-to-end fashion. We instantiate this framework with
SUmmarization augmented Policy Optimization
(SUPO), an LLM RL algorithm that enables long-horizon training beyond
a fixed context limit. Experiments on interactive function calling and
searching tasks demonstrate that SUPO significantly improves the
success rate while maintaining the same or even lower working context length
compared to baselines. We also demonstrate that for complex searching tasks,
SUPO can further improve the evaluation performance when scaling
test-time maximum round of summarization beyond that of training time. Our
results establish summarization-based context management as a principled and
scalable approach for training RL agents beyond a fixed context length limit.
Summarization-based context management in reinforcement learning fine-tunes large language model agents for long-horizon tool use, improving success rates and scalability beyond fixed context limits.
reinforcement learning large language model long-horizon multi-turn tool use context length summarization-based context management policy gradient SUPO interactive function calling searching tasks