J1: Incentivizing Thinking in LLM-as-a-Judge via Reinforcement Learning
The progress of AI is bottlenecked by the quality of evaluation, and powerful
LLM-as-a-Judge models have proved to be a core solution. Improved judgment
ability is enabled by stronger chain-of-thought reasoning, motivating the need
to find the best recipes for training such models to think. In this work we
introduce J1, a reinforcement learning approach to training such models. Our
method converts both verifiable and non-verifiable prompts to judgment tasks
with verifiable rewards that incentivize thinking and mitigate judgment bias.
In particular, our approach outperforms all other existing 8B or 70B models
when trained at those sizes, including models distilled from DeepSeek-R1. J1
also outperforms o1-mini, and even R1 on some benchmarks, despite training a
smaller model. We provide analysis and ablations comparing Pairwise-J1 vs
Pointwise-J1 models, offline vs online training recipes, reward strategies,
seed prompts, and variations in thought length and content. We find that our
models make better judgments by learning to outline evaluation criteria,
comparing against self-generated reference answers, and re-evaluating the
correctness of model responses.
6826a180cad9000ebc70f038
Chenxi Whitehouse
6826a180cad9000ebc70f039
Tianlu Wang
admin_assigned
2025-05-16T13:42:23.084Z
63f6a0e6b4c9a104f4b95d86
/avatars/660c3e44bea976f3e0560bf95466b03c.svg
Tianlu Wang
user
Tianlu
6826a180cad9000ebc70f03a
Ping Yu
6826a180cad9000ebc70f03b
Xian Li
6826a180cad9000ebc70f03c
Jason Weston
admin_assigned
2025-05-16T13:42:55.728Z
62f023a36a027498eaa2f9cc
/avatars/8ac1c5c74d0957e3c6cc94b3a7795c37.svg
Jason Weston
user
spermwhale
6826a180cad9000ebc70f03d
Ilia Kulikov
admin_assigned
2025-05-16T13:43:03.749Z
65d645913bc37244c55c39d5
/avatars/508372876a654957650ee9429886d10c.svg
ilia kulikov
user
uralik
6826a180cad9000ebc70f03e
Swarnadeep Saha
extracted_pending
2025-05-16T02:22:57.318Z
64b75f4b037d6452a30f71aa
/avatars/5a0322e7ecda05164e45526d605e3619.svg
Swarnadeep Saha
user
swarna92
J1: Incentivizing Thinking in LLM-as-a-Judge via Reinforcement Learning
The progress of AI is bottlenecked by the quality of evaluation, and powerful
LLM-as-a-Judge models have proved to be a core solution. Improved judgment
ability is enabled by stronger chain-of-thought reasoning, motivating the need
to find the best recipes for training such models to think. In this work we
introduce J1, a reinforcement learning approach to training such models. Our
method converts both verifiable and non-verifiable prompts to judgment tasks
with verifiable rewards that incentivize thinking and mitigate judgment bias.
In particular, our approach outperforms all other existing 8B or 70B models
when trained at those sizes, including models distilled from DeepSeek-R1. J1
also outperforms o1-mini, and even R1 on some benchmarks, despite training a
smaller model. We provide analysis and ablations comparing Pairwise-J1 vs
Pointwise-J1 models, offline vs online training recipes, reward strategies,
seed prompts, and variations in thought length and content. We find that our
models make better judgments by learning to outline evaluation criteria,
comparing against self-generated reference answers, and re-evaluating the
correctness of model responses.
A reinforcement learning approach called J1 improves the judgment ability of LLM-as-a-Judge models through verifiable rewards and chain-of-thought reasoning.
reinforcement learning chain-of-thought reasoning judgment tasks reward strategies offline training online training Pairwise-J1 Pointwise-J1 evaluation criteria self-generated reference answers re-evaluation