Group-Relative REINFORCE Is Secretly an Off-Policy Algorithm:
  Demystifying Some Myths About GRPO and Its Friends
Off-policy reinforcement learning (RL) for large language models (LLMs) is
attracting growing interest, driven by practical constraints in real-world
applications, the complexity of LLM-RL infrastructure, and the need for further
innovations of RL methodologies. While classic REINFORCE and its modern
variants like Group Relative Policy Optimization (GRPO) are typically regarded
as on-policy algorithms with limited tolerance of off-policyness, we present in
this work a first-principles derivation for group-relative REINFORCE without
assuming a specific training data distribution, showing that it admits a native
off-policy interpretation. This perspective yields two general principles for
adapting REINFORCE to off-policy settings: regularizing policy updates, and
actively shaping the data distribution. Our analysis demystifies some myths
about the roles of importance sampling and clipping in GRPO, unifies and
reinterprets two recent algorithms -- Online Policy Mirror Descent (OPMD) and
Asymmetric REINFORCE (AsymRE) -- as regularized forms of the REINFORCE loss,
and offers theoretical justification for seemingly heuristic data-weighting
strategies. Our findings lead to actionable insights that are validated with
extensive empirical studies, and open up new opportunities for principled
algorithm design in off-policy RL for LLMs. Source code for this work is
available at
https://github.com/modelscope/Trinity-RFT/tree/main/examples/rec_gsm8k.
68df35c1df49fb0df1e03cb3
Chaorui Yao
68df35c1df49fb0df1e03cb4
Yanxi Chen
claimed_verified
2025-10-03T12:29:49.553Z
6576f9f4654561a1b345610b
/avatars/f801f551640caa70368fcc26a0f51d27.svg
Yanxi Chen
user
yanxi-chen
68df35c1df49fb0df1e03cb5
Yuchang Sun
68df35c1df49fb0df1e03cb6
Yushuo Chen
68df35c1df49fb0df1e03cb7
Wenhao Zhang
68df35c1df49fb0df1e03cb8
Xuchen Pan
68df35c1df49fb0df1e03cb9
Yaliang Li
68df35c1df49fb0df1e03cba
Bolin Ding
Group-Relative REINFORCE Is Secretly an Off-Policy Algorithm:
  Demystifying Some Myths About GRPO and Its Friends
Off-policy reinforcement learning (RL) for large language models (LLMs) is
attracting growing interest, driven by practical constraints in real-world
applications, the complexity of LLM-RL infrastructure, and the need for further
innovations of RL methodologies. While classic REINFORCE and its modern
variants like Group Relative Policy Optimization (GRPO) are typically regarded
as on-policy algorithms with limited tolerance of off-policyness, we present in
this work a first-principles derivation for group-relative REINFORCE without
assuming a specific training data distribution, showing that it admits a native
off-policy interpretation. This perspective yields two general principles for
adapting REINFORCE to off-policy settings: regularizing policy updates, and
actively shaping the data distribution. Our analysis demystifies some myths
about the roles of importance sampling and clipping in GRPO, unifies and
reinterprets two recent algorithms -- Online Policy Mirror Descent (OPMD) and
Asymmetric REINFORCE (AsymRE) -- as regularized forms of the REINFORCE loss,
and offers theoretical justification for seemingly heuristic data-weighting
strategies. Our findings lead to actionable insights that are validated with
extensive empirical studies, and open up new opportunities for principled
algorithm design in off-policy RL for LLMs. Source code for this work is
available at
https://github.com/modelscope/Trinity-RFT/tree/main/examples/rec_gsm8k.
Off-policy reinforcement learning for large language models is explored through a new derivation of group-relative REINFORCE, offering insights into importance sampling, clipping, and data-weighting strategies.
off-policy reinforcement learning large language models REINFORCE Group Relative Policy Optimization GRPO Online Policy Mirror Descent OPMD Asymmetric REINFORCE AsymRE importance sampling clipping data-weighting strategies