On-Policy RL Meets Off-Policy Experts: Harmonizing Supervised
  Fine-Tuning and Reinforcement Learning via Dynamic Weighting
Supervised Fine-Tuning (SFT) and Reinforcement Learning (RL) are two
prominent post-training paradigms for refining the capabilities and aligning
the behavior of Large Language Models (LLMs). Existing approaches that
integrate SFT and RL often face the risk of disrupting established model
patterns and inducing overfitting to expert data. To address this, we present a
novel investigation into the unified view of SFT and RL through an off-policy
versus on-policy lens. We propose CHORD, a framework for the Controllable
Harmonization of On- and Off-Policy Reinforcement Learning via Dynamic
Weighting, which reframes SFT not as a separate stage but as a dynamically
weighted auxiliary objective within the on-policy RL process. Based on an
analysis of off-policy expert data's influence at both holistic and granular
levels, we incorporate a dual-control mechanism in CHORD. Specifically, the
framework first employs a global coefficient to holistically guide the
transition from off-policy imitation to on-policy exploration, and then applies
a token-wise weighting function that enables granular learning from expert
tokens, which preserves on-policy exploration and mitigates disruption from
off-policy data. We conduct extensive experiments on widely used benchmarks,
providing empirical evidence that CHORD achieves a stable and efficient
learning process. By effectively harmonizing off-policy expert data with
on-policy exploration, CHORD demonstrates significant improvements over
baselines. We release the implementation at
https://github.com/modelscope/Trinity-RFT/tree/main/examples/mix_chord to
inspire further research.
68a52f576cf0bf898542ec27
Wenhao Zhang
claimed_verified
2025-08-20T08:51:54.803Z
63f46a0fa096536aeab6ee75
https://cdn-avatars.huggingface.co/v1/production/uploads/63f46a0fa096536aeab6ee75/Bte71XHp05z_vXvt8POev.png
garyzhang
user
xiaoniqiu
68a52f576cf0bf898542ec28
Yuexiang Xie
68a52f576cf0bf898542ec29
Yuchang Sun
68a52f576cf0bf898542ec2a
Yanxi Chen
68a52f576cf0bf898542ec2b
Guoyin Wang
68a52f576cf0bf898542ec2c
Yaliang Li
68a52f576cf0bf898542ec2d
Bolin Ding
68a52f576cf0bf898542ec2e
Jingren Zhou
On-Policy RL Meets Off-Policy Experts: Harmonizing Supervised
  Fine-Tuning and Reinforcement Learning via Dynamic Weighting
Supervised Fine-Tuning (SFT) and Reinforcement Learning (RL) are two
prominent post-training paradigms for refining the capabilities and aligning
the behavior of Large Language Models (LLMs). Existing approaches that
integrate SFT and RL often face the risk of disrupting established model
patterns and inducing overfitting to expert data. To address this, we present a
novel investigation into the unified view of SFT and RL through an off-policy
versus on-policy lens. We propose CHORD, a framework for the Controllable
Harmonization of On- and Off-Policy Reinforcement Learning via Dynamic
Weighting, which reframes SFT not as a separate stage but as a dynamically
weighted auxiliary objective within the on-policy RL process. Based on an
analysis of off-policy expert data's influence at both holistic and granular
levels, we incorporate a dual-control mechanism in CHORD. Specifically, the
framework first employs a global coefficient to holistically guide the
transition from off-policy imitation to on-policy exploration, and then applies
a token-wise weighting function that enables granular learning from expert
tokens, which preserves on-policy exploration and mitigates disruption from
off-policy data. We conduct extensive experiments on widely used benchmarks,
providing empirical evidence that CHORD achieves a stable and efficient
learning process. By effectively harmonizing off-policy expert data with
on-policy exploration, CHORD demonstrates significant improvements over
baselines. We release the implementation at
https://github.com/modelscope/Trinity-RFT/tree/main/examples/mix_chord to
inspire further research.
CHORD integrates supervised fine-tuning and reinforcement learning by dynamically weighting off-policy and on-policy data to improve model stability and performance.
Supervised Fine-Tuning Reinforcement Learning Large Language Models off-policy on-policy Controllable Harmonization Dynamic Weighting dual-control mechanism global coefficient token-wise weighting function on-policy exploration off-policy data empirical evidence benchmarks