Temporal Self-Rewarding Language Models: Decoupling Chosen-Rejected via
  Past-Future
Self-Rewarding Language Models propose an architecture in which the Large
Language Models(LLMs) both generates responses and evaluates its own outputs
via LLM-as-a-Judge prompting, dynamically improving its generative capabilities
through iterative Direct Preference Optimization (DPO). However, our analysis
reveals a critical limitation in existing Self-Rewarding paradigms: the
synchronized improvement of chosen and rejected responses progressively narrows
the representational difference between contrasting samples, undermining
effective preference learning. We propose Temporal Self-Rewarding
Language Models that strategically coordinate past, present, and future model
generations to sustain learning signals. Our dual-phase framework introduces:
(1) Anchored Rejection - fixing rejected responses using the past
initial model's outputs and (2) Future-Guided Chosen - dynamically
curating chosen samples using next-generation model predictions. Extensive
experiments across three model families (Llama, Qwen, Mistral) and different
model sizes (Llama3B/8B/70B) demonstrate significant improvements when trained
with our method compared to Self-Rewarding using same computation resources.
For example, Llama3.1-8B reaches a 29.44 win rate on AlpacaEval 2.0 with our
method, outperforming the Self-Rewarding baseline (19.69) by 9.75. Notably, our
method also demonstrates superior out-of-distribution generalization across
mathematical reasoning (GSM8K), knowledge-based QA (ARC, TruthfulQA), and code
generation (HumanEval) tasks, even though we do not specifically collect such
training data.
689abd09fab6fdd2e52ac4c4
Yidong Wang
689abd09fab6fdd2e52ac4c5
Xin Wang
claimed_verified
2025-08-13T07:19:52.114Z
66bdae84b8a029b4fdf81eb4
/avatars/779b9d9f307f34759b18781208b5b1a4.svg
xin wang
user
xin121
689abd09fab6fdd2e52ac4c6
Cunxiang Wang
689abd09fab6fdd2e52ac4c7
Junfeng Fang
689abd09fab6fdd2e52ac4c8
Qiufeng Wang
689abd09fab6fdd2e52ac4c9
Jianing Chu
689abd09fab6fdd2e52ac4ca
Xuran Meng
689abd09fab6fdd2e52ac4cb
Shuxun Yang
689abd09fab6fdd2e52ac4cc
Libo Qin
689abd09fab6fdd2e52ac4cd
Yue Zhang
689abd09fab6fdd2e52ac4ce
Wei Ye
689abd09fab6fdd2e52ac4cf
Shikun Zhang
Temporal Self-Rewarding Language Models: Decoupling Chosen-Rejected via
  Past-Future
Self-Rewarding Language Models propose an architecture in which the Large
Language Models(LLMs) both generates responses and evaluates its own outputs
via LLM-as-a-Judge prompting, dynamically improving its generative capabilities
through iterative Direct Preference Optimization (DPO). However, our analysis
reveals a critical limitation in existing Self-Rewarding paradigms: the
synchronized improvement of chosen and rejected responses progressively narrows
the representational difference between contrasting samples, undermining
effective preference learning. We propose Temporal Self-Rewarding
Language Models that strategically coordinate past, present, and future model
generations to sustain learning signals. Our dual-phase framework introduces:
(1) Anchored Rejection - fixing rejected responses using the past
initial model's outputs and (2) Future-Guided Chosen - dynamically
curating chosen samples using next-generation model predictions. Extensive
experiments across three model families (Llama, Qwen, Mistral) and different
model sizes (Llama3B/8B/70B) demonstrate significant improvements when trained
with our method compared to Self-Rewarding using same computation resources.
For example, Llama3.1-8B reaches a 29.44 win rate on AlpacaEval 2.0 with our
method, outperforming the Self-Rewarding baseline (19.69) by 9.75. Notably, our
method also demonstrates superior out-of-distribution generalization across
mathematical reasoning (GSM8K), knowledge-based QA (ARC, TruthfulQA), and code
generation (HumanEval) tasks, even though we do not specifically collect such
training data.
Temporal Self-Rewarding Language Models improve generative capabilities by strategically using past and future model outputs, enhancing preference learning and out-of-distribution generalization.
Self-Rewarding Language Models Large Language Models LLM-as-a-Judge Direct Preference Optimization Anchored Rejection Future-Guided Chosen Temporal Self-Rewarding Language Models AlpacaEval GSM8K ARC TruthfulQA HumanEval