Training-Free Group Relative Policy Optimization
Recent advances in Large Language Model (LLM) agents have demonstrated their
promising general capabilities. However, their performance in specialized
real-world domains often degrades due to challenges in effectively integrating
external tools and specific prompting strategies. While methods like agentic
reinforcement learning have been proposed to address this, they typically rely
on costly parameter updates, for example, through a process that uses
Supervised Fine-Tuning (SFT) followed by a Reinforcement Learning (RL) phase
with Group Relative Policy Optimization (GRPO) to alter the output
distribution. However, we argue that LLMs can achieve a similar effect on the
output distribution by learning experiential knowledge as a token prior, which
is a far more lightweight approach that not only addresses practical data
scarcity but also avoids the common issue of overfitting. To this end, we
propose Training-Free Group Relative Policy Optimization (Training-Free GRPO),
a cost-effective solution that enhances LLM agent performance without any
parameter updates. Our method leverages the group relative semantic advantage
instead of numerical ones within each group of rollouts, iteratively distilling
high-quality experiential knowledge during multi-epoch learning on a minimal
ground-truth data. Such knowledge serves as the learned token prior, which is
seamlessly integrated during LLM API calls to guide model behavior. Experiments
on mathematical reasoning and web searching tasks demonstrate that
Training-Free GRPO, when applied to DeepSeek-V3.1-Terminus, significantly
improves out-of-domain performance. With just a few dozen training samples,
Training-Free GRPO outperforms fine-tuned small LLMs with marginal training
data and cost.
68e8725295e8e6771df389aa
Yuzheng Cai
claimed_verified
2025-10-10T08:58:05.142Z
66e258bdc70c02b46dfed6e3
/avatars/ccc2d604616c018f45a268a610472cac.svg
Yuzheng Cai
user
Ucreate
68e8725295e8e6771df389ab
Siqi Cai
68e8725295e8e6771df389ac
Yuchen Shi
68e8725295e8e6771df389ad
Zihan Xu
68e8725295e8e6771df389ae
Lichao Chen
68e8725295e8e6771df389af
Yulei Qin
claimed_verified
2025-10-10T04:32:09.755Z
6390525c00fb8ec4a424e0ff
/avatars/4302571e2ef4a9875491221aa630a329.svg
Yulei Qin
user
yolay
68e8725295e8e6771df389b0
Xiaoyu Tan
68e8725295e8e6771df389b1
Gang Li
68e8725295e8e6771df389b2
Zongyi Li
68e8725295e8e6771df389b3
Haojia Lin
68e8725295e8e6771df389b4
Yong Mao
68e8725295e8e6771df389b5
Ke Li
68e8725295e8e6771df389b6
Xing Sun
claimed_verified
2025-10-11T13:48:07.194Z
647401e50da364bd0d002f2a
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/vPuPn7EV092mLBOM2YZXd.png
XING SUN
user
tedsun
Training-Free Group Relative Policy Optimization
Recent advances in Large Language Model (LLM) agents have demonstrated their
promising general capabilities. However, their performance in specialized
real-world domains often degrades due to challenges in effectively integrating
external tools and specific prompting strategies. While methods like agentic
reinforcement learning have been proposed to address this, they typically rely
on costly parameter updates, for example, through a process that uses
Supervised Fine-Tuning (SFT) followed by a Reinforcement Learning (RL) phase
with Group Relative Policy Optimization (GRPO) to alter the output
distribution. However, we argue that LLMs can achieve a similar effect on the
output distribution by learning experiential knowledge as a token prior, which
is a far more lightweight approach that not only addresses practical data
scarcity but also avoids the common issue of overfitting. To this end, we
propose Training-Free Group Relative Policy Optimization (Training-Free GRPO),
a cost-effective solution that enhances LLM agent performance without any
parameter updates. Our method leverages the group relative semantic advantage
instead of numerical ones within each group of rollouts, iteratively distilling
high-quality experiential knowledge during multi-epoch learning on a minimal
ground-truth data. Such knowledge serves as the learned token prior, which is
seamlessly integrated during LLM API calls to guide model behavior. Experiments
on mathematical reasoning and web searching tasks demonstrate that
Training-Free GRPO, when applied to DeepSeek-V3.1-Terminus, significantly
improves out-of-domain performance. With just a few dozen training samples,
Training-Free GRPO outperforms fine-tuned small LLMs with marginal training
data and cost.
Training-Free GRPO enhances LLM agent performance in specialized domains by learning experiential knowledge as a token prior without parameter updates, improving out-of-domain tasks with minimal data.
Large Language Model (LLM) agentic reinforcement learning Supervised Fine-Tuning (SFT) Reinforcement Learning (RL) Group Relative Policy Optimization (GRPO) token prior Training-Free GRPO group relative semantic advantage multi-epoch learning minimal ground-truth data DeepSeek-V3.1-Terminus mathematical reasoning web searching tasks