Teaching Large Language Models to Reason with Reinforcement Learning
Reinforcement Learning from Human Feedback (RLHF) has emerged as a
dominant approach for aligning LLM outputs with human preferences. Inspired by
the success of RLHF, we study the performance of multiple algorithms that learn
from feedback (Expert Iteration, Proximal Policy Optimization (PPO),
Return-Conditioned RL) on improving LLM reasoning capabilities. We investigate
both sparse and dense rewards provided to the LLM both heuristically and via a
learned reward model. We additionally start from multiple model sizes and
initializations both with and without supervised fine-tuning (SFT)
data. Overall, we find all algorithms perform comparably, with Expert Iteration
performing best in most cases. Surprisingly, we find the sample complexity of
Expert Iteration is similar to that of PPO, requiring at most on the order of
10^6 samples to converge from a pretrained checkpoint. We investigate why
this is the case, concluding that during RL training models fail to explore
significantly beyond solutions already produced by SFT models. Additionally, we
discuss a trade off between maj@1 and pass@96 metric performance during SFT
training and how conversely RL training improves both simultaneously. We then
conclude by discussing the implications of our findings for RLHF and the future
role of RL in LLM fine-tuning.
65ea7cba8d4052d338ab202c
Alex Havrilla
admin_assigned
2024-03-08T09:13:42.441Z
6144d541fa30a4ba3ef7ef15
/avatars/eeb849eb29e249fd1b40e5c10d9bf9e5.svg
Alex Havrilla
user
Dahoas
65ea7cba8d4052d338ab202d
Yuqing Du
admin_assigned
2024-03-08T09:13:49.773Z
64b7915aa8c39dc07887e035
/avatars/271bd18eb7d4b5bae8d71674ccb3a792.svg
Yuqing Du
user
yuqingd
65ea7cba8d4052d338ab202e
Sharath Chandra Raparthy
admin_assigned
2024-03-08T09:14:01.734Z
65007a96ea5699b59a9bb432
/avatars/4549f5964b007de189c45a77daf10075.svg
Sharath Raparthy
user
sharathraparthy
65ea7cba8d4052d338ab202f
Christoforos Nalmpantis
admin_assigned
2024-03-08T09:14:08.341Z
627e1e4f9632ca4815f0a729
/avatars/2ba615cdc1d9e3eb0397366cff7a7c8b.svg
Christoforos Nalmpantis
user
christofernal
65ea7cba8d4052d338ab2030
Jane Dwivedi-Yu
65ea7cba8d4052d338ab2031
Maksym Zhuravinskyi
65ea7cba8d4052d338ab2032
Eric Hambro
65ea7cba8d4052d338ab2033
Sainbayar Sukhbaatar
claimed_verified
2024-07-30T16:01:24.362Z
66a8611eb51510d82ed54231
/avatars/ad559e774fee4914091b82c9831ae2a2.svg
Sainbayar Sukhbaatar
user
sainbar
65ea7cba8d4052d338ab2034
Roberta Raileanu
admin_assigned
2024-03-08T09:14:42.769Z
633e94793a17ab61de8e2b9c
/avatars/5f2f58ddeed211393660ada6b135f0d5.svg
Roberta Raileanu
user
rraileanu
Teaching Large Language Models to Reason with Reinforcement Learning
Reinforcement Learning from Human Feedback (RLHF) has emerged as a
dominant approach for aligning LLM outputs with human preferences. Inspired by
the success of RLHF, we study the performance of multiple algorithms that learn
from feedback (Expert Iteration, Proximal Policy Optimization (PPO),
Return-Conditioned RL) on improving LLM reasoning capabilities. We investigate
both sparse and dense rewards provided to the LLM both heuristically and via a
learned reward model. We additionally start from multiple model sizes and
initializations both with and without supervised fine-tuning (SFT)
data. Overall, we find all algorithms perform comparably, with Expert Iteration
performing best in most cases. Surprisingly, we find the sample complexity of
Expert Iteration is similar to that of PPO, requiring at most on the order of
10^6 samples to converge from a pretrained checkpoint. We investigate why
this is the case, concluding that during RL training models fail to explore
significantly beyond solutions already produced by SFT models. Additionally, we
discuss a trade off between maj@1 and pass@96 metric performance during SFT
training and how conversely RL training improves both simultaneously. We then
conclude by discussing the implications of our findings for RLHF and the future
role of RL in LLM fine-tuning.
Multiple RLHF algorithms, including Expert Iteration, PPO, and Return-Conditioned RL, similarly improve LLM reasoning with similar sample complexity, with Expert Iteration slightly outperforming others.
Reinforcement Learning from Human Feedback RLHF Expert Iteration Proximal Policy Optimization PPO Return-Conditioned RL sparse rewards dense rewards reward model model sizes initializations supervised fine-tuning SFT sample complexity pretrained checkpoint maj@1 pass@96