Towards a Unified View of Large Language Model Post-Training
Two major sources of training data exist for post-training modern language
models: online (model-generated rollouts) data, and offline (human or
other-model demonstrations) data. These two types of data are typically used by
approaches like Reinforcement Learning (RL) and Supervised Fine-Tuning (SFT),
respectively. In this paper, we show that these approaches are not in
contradiction, but are instances of a single optimization process. We derive a
Unified Policy Gradient Estimator, and present the calculations of a wide
spectrum of post-training approaches as the gradient of a common objective
under different data distribution assumptions and various bias-variance
tradeoffs. The gradient estimator is constructed with four interchangeable
parts: stabilization mask, reference policy denominator, advantage estimate,
and likelihood gradient. Motivated by our theoretical findings, we propose
Hybrid Post-Training (HPT), an algorithm that dynamically selects different
training signals. HPT is designed to yield both effective exploitation of
demonstration and stable exploration without sacrificing learned reasoning
patterns. We provide extensive experiments and ablation studies to verify the
effectiveness of our unified theoretical framework and HPT. Across six
mathematical reasoning benchmarks and two out-of-distribution suites, HPT
consistently surpasses strong baselines across models of varying scales and
families.
68ba4a27736018af705e8e47
Xingtai Lv
claimed_verified
2025-09-05T15:21:46.656Z
663f07d029be04778ba97871
/avatars/fb7c9d4a2c537d918a3267e7cbc03f04.svg
Xingtai Lv
user
XingtaiHF
68ba4a27736018af705e8e48
Yuxin Zuo
claimed_verified
2025-09-12T16:21:08.482Z
622474f38dc6b0b64f5e903d
/avatars/d6b60a014277a8ec7d564163c5f644aa.svg
Yuxin Zuo
user
yuxinzuo
68ba4a27736018af705e8e49
Youbang Sun
claimed_verified
2025-09-06T10:57:12.954Z
679ce8c048ebd7903d76a832
/avatars/5f3fecaacfee6e2d5a72dd19fe87055a.svg
Youbang Sun
user
Youbang
68ba4a27736018af705e8e4a
Hongyi Liu
claimed_verified
2025-09-08T07:02:31.137Z
66d12f904cc92b9866c7f837
/avatars/5c3ce46c1f953a7081e344cd91a8d17b.svg
Hongyi Liu
user
hongyi-liu
68ba4a27736018af705e8e4b
Yuntian Wei
claimed_verified
2025-09-06T10:57:10.829Z
686d04f0789ffa7e032daa1d
/avatars/e129520cf1f03966db1f066d685b446b.svg
Jason Wei
user
Y-T-Wei
68ba4a27736018af705e8e4c
Zhekai Chen
68ba4a27736018af705e8e4d
Lixuan He
68ba4a27736018af705e8e4e
Xuekai Zhu
claimed_verified
2025-09-19T06:50:44.476Z
647ffddeb82adfa7cc1a10d9
/avatars/26aa168d6b2068298ebb16584aa52b6c.svg
zhu
user
xuekai
68ba4a27736018af705e8e4f
Kaiyan Zhang
claimed_verified
2025-09-05T08:01:42.328Z
60bc94cd85a3ab33829b6211
/avatars/b57d36c7577fbbb42ea5b963eef4144a.svg
Kaiyan Zhang
user
iseesaw
68ba4a27736018af705e8e50
Bingning Wang
68ba4a27736018af705e8e51
Ning Ding
claimed_verified
2025-09-13T14:56:51.040Z
60cf4bcb1ce3775ebb86e5d5
/avatars/12bcd18d215abf91f297f93007733148.svg
Ning Ding
user
stingning
68ba4a27736018af705e8e52
Bowen Zhou
Towards a Unified View of Large Language Model Post-Training
Two major sources of training data exist for post-training modern language
models: online (model-generated rollouts) data, and offline (human or
other-model demonstrations) data. These two types of data are typically used by
approaches like Reinforcement Learning (RL) and Supervised Fine-Tuning (SFT),
respectively. In this paper, we show that these approaches are not in
contradiction, but are instances of a single optimization process. We derive a
Unified Policy Gradient Estimator, and present the calculations of a wide
spectrum of post-training approaches as the gradient of a common objective
under different data distribution assumptions and various bias-variance
tradeoffs. The gradient estimator is constructed with four interchangeable
parts: stabilization mask, reference policy denominator, advantage estimate,
and likelihood gradient. Motivated by our theoretical findings, we propose
Hybrid Post-Training (HPT), an algorithm that dynamically selects different
training signals. HPT is designed to yield both effective exploitation of
demonstration and stable exploration without sacrificing learned reasoning
patterns. We provide extensive experiments and ablation studies to verify the
effectiveness of our unified theoretical framework and HPT. Across six
mathematical reasoning benchmarks and two out-of-distribution suites, HPT
consistently surpasses strong baselines across models of varying scales and
families.
A unified policy gradient estimator and Hybrid Post-Training algorithm effectively combine online and offline data for post-training language models, improving performance across various benchmarks.
Reinforcement Learning Supervised Fine-Tuning Unified Policy Gradient Estimator stabilization mask reference policy denominator advantage estimate likelihood gradient Hybrid Post-Training mathematical reasoning benchmarks out-of-distribution suites