Quantile Reward Policy Optimization: Alignment with Pointwise Regression and Exact Partition Functions
Simon Matrenok, Skander Moalla, Caglar Gulcehre
We introduce Quantile Reward Policy Optimization (QRPO), an alignment algorithm for large language models (LLM) that integrates the simplicity and offline efficiency of policy fitting methods, such as DPO and REBEL, with the ability to better leverage the signal of pointwise absolute rewards, previously limited to only policy improvement approaches such as PPO and GRPO. QRPO regresses to the closed-form solution of the KL-regularized RL objective using the quantile reward, which we show makes the partition function analytically tractable, and thus overcomes the need to resort to preferences or relative rewards to cancel this term. QRPO consistently provides the best results on chats evaluated on AlpacaEval 2 and coding evaluated on LeetCode compared to DPO, REBEL, and SimPO trained with different datasets and models on the 8B scale. We observe that training with rewards instead of preferences induces less length bias in the learned models.  Finally, we show that QRPO scales with more compute budget to estimate the quantile rewards, opening a new dimension for pre-computation scaling.