Reinforcing Multi-Turn Reasoning in LLM Agents via Turn-Level Credit
  Assignment
This paper investigates approaches to enhance the reasoning capabilities of
Large Language Model (LLM) agents using Reinforcement Learning (RL).
Specifically, we focus on multi-turn tool-use scenarios, which can be naturally
modeled as Markov Decision Processes (MDPs). While existing approaches often
train multi-turn LLM agents with trajectory-level advantage estimation in
bandit settings, they struggle with turn-level credit assignment across
multiple decision steps, limiting their performance on multi-turn reasoning
tasks. To address this, we introduce a fine-grained turn-level advantage
estimation strategy to enable more precise credit assignment in multi-turn
agent interactions. The strategy is general and can be incorporated into
various RL algorithms such as Group Relative Preference Optimization (GRPO).
Our experimental evaluation on multi-turn reasoning and search-based tool-use
tasks with GRPO implementations highlights the effectiveness of the MDP
framework and the turn-level credit assignment in advancing the multi-turn
reasoning capabilities of LLM agents in complex decision-making settings. Our
method achieves 100% success in tool execution and 50% accuracy in exact answer
matching, significantly outperforming baselines, which fail to invoke tools and
achieve only 20-30% exact match accuracy.
6832def11f8965bd58214baf
Siliang Zeng
6832def11f8965bd58214bb0
Quan Wei
claimed_verified
2025-05-29T20:18:40.227Z
662a037140363679757cc292
/avatars/fc41455247cc0744f24ac9255293ddf5.svg
Quan Wei
user
quanwei0
6832def11f8965bd58214bb1
William Brown
6832def11f8965bd58214bb2
Oana Frunza
6832def11f8965bd58214bb3
Yuriy Nevmyvaka
6832def11f8965bd58214bb4
Mingyi Hong
claimed_verified
2025-06-02T07:49:23.598Z
679a5fa807076ced8ae5f41b
/avatars/63cac0a05c9c7500c9ab9c75667a38d7.svg
Mingyi Hong
user
poborskypaul
Reinforcing Multi-Turn Reasoning in LLM Agents via Turn-Level Credit
  Assignment
This paper investigates approaches to enhance the reasoning capabilities of
Large Language Model (LLM) agents using Reinforcement Learning (RL).
Specifically, we focus on multi-turn tool-use scenarios, which can be naturally
modeled as Markov Decision Processes (MDPs). While existing approaches often
train multi-turn LLM agents with trajectory-level advantage estimation in
bandit settings, they struggle with turn-level credit assignment across
multiple decision steps, limiting their performance on multi-turn reasoning
tasks. To address this, we introduce a fine-grained turn-level advantage
estimation strategy to enable more precise credit assignment in multi-turn
agent interactions. The strategy is general and can be incorporated into
various RL algorithms such as Group Relative Preference Optimization (GRPO).
Our experimental evaluation on multi-turn reasoning and search-based tool-use
tasks with GRPO implementations highlights the effectiveness of the MDP
framework and the turn-level credit assignment in advancing the multi-turn
reasoning capabilities of LLM agents in complex decision-making settings. Our
method achieves 100% success in tool execution and 50% accuracy in exact answer
matching, significantly outperforming baselines, which fail to invoke tools and
achieve only 20-30% exact match accuracy.
Reinforcement Learning with turn-level credit assignment enhances Large Language Model reasoning capabilities in multi-turn tool-use scenarios.
Large Language Model Reinforcement Learning multi-turn tool-use Markov Decision Processes trajectory-level advantage estimation turn-level credit assignment Group Relative Preference Optimization multi-turn reasoning search-based tool-use tool execution exact answer matching