Process-based Self-Rewarding Language Models
Large Language Models have demonstrated outstanding performance across
various downstream tasks and have been widely applied in multiple scenarios.
Human-annotated preference data is used for training to further improve LLMs'
performance, which is constrained by the upper limit of human performance.
Therefore, Self-Rewarding method has been proposed, where LLMs generate
training data by rewarding their own outputs. However, the existing
self-rewarding paradigm is not effective in mathematical reasoning scenarios
and may even lead to a decline in performance. In this work, we propose the
Process-based Self-Rewarding pipeline for language models, which introduces
long-thought reasoning, step-wise LLM-as-a-Judge, and step-wise preference
optimization within the self-rewarding paradigm. Our new paradigm successfully
enhances the performance of LLMs on multiple mathematical reasoning benchmarks
through iterative Process-based Self-Rewarding, demonstrating the immense
potential of self-rewarding to achieve LLM reasoning that may surpass human
capabilities.
67c991abe7dba9cdf9f41e67
Shimao Zhang
claimed_verified
2025-03-06T19:50:45.146Z
65080dc63fc966d1bbba485d
/avatars/347890233f2316e7f7a04d652b2378bb.svg
Shimao Zhang
user
Shimao-Zhang
67c991abe7dba9cdf9f41e68
Xiao Liu
extracted_confirmed
2025-03-06T12:14:52.016Z
63fb6e281b4b1bd4e7ffc5be
https://cdn-avatars.huggingface.co/v1/production/uploads/63fb6e281b4b1bd4e7ffc5be/aiRu_bulgnxvEMrjipGoQ.jpeg
Xiao Liu
user
lx865712528
67c991abe7dba9cdf9f41e69
Xin Zhang
67c991abe7dba9cdf9f41e6a
Junxiao Liu
claimed_verified
2025-05-27T07:56:21.963Z
68356f5db243fb809813a715
https://cdn-avatars.huggingface.co/v1/production/uploads/68356f5db243fb809813a715/grhHvANfDRp75rMJxWlQo.jpeg
LiuJunxiao
user
master-lan
67c991abe7dba9cdf9f41e6b
Zheheng Luo
67c991abe7dba9cdf9f41e6c
Shujian Huang
67c991abe7dba9cdf9f41e6d
Yeyun Gong
Process-based Self-Rewarding Language Models
Large Language Models have demonstrated outstanding performance across
various downstream tasks and have been widely applied in multiple scenarios.
Human-annotated preference data is used for training to further improve LLMs'
performance, which is constrained by the upper limit of human performance.
Therefore, Self-Rewarding method has been proposed, where LLMs generate
training data by rewarding their own outputs. However, the existing
self-rewarding paradigm is not effective in mathematical reasoning scenarios
and may even lead to a decline in performance. In this work, we propose the
Process-based Self-Rewarding pipeline for language models, which introduces
long-thought reasoning, step-wise LLM-as-a-Judge, and step-wise preference
optimization within the self-rewarding paradigm. Our new paradigm successfully
enhances the performance of LLMs on multiple mathematical reasoning benchmarks
through iterative Process-based Self-Rewarding, demonstrating the immense
potential of self-rewarding to achieve LLM reasoning that may surpass human
capabilities.
A Process-based Self-Rewarding pipeline enhances LLMs' mathematical reasoning by iteratively evaluating and optimizing their own outputs.
Large Language Models self-rewarding LLM-as-a-Judge step-wise preference optimization mathematical reasoning Process-based Self-Rewarding