CompassJudger-2: Towards Generalist Judge Model via Verifiable Rewards
Recently, the role of LLM-as-judge in evaluating large language models has
gained prominence. However, current judge models suffer from narrow
specialization and limited robustness, undermining their capacity for
comprehensive evaluations. In this work, we present CompassJudger-2, a novel
generalist judge model that overcomes these limitations via a task-driven,
multi-domain data curation strategy. Central to our approach is supervising
judgment tasks with verifiable rewards, guiding intrinsic critical reasoning
through rejection sampling to foster robust, generalizable judgment
capabilities. We introduce a refined learning objective with margin policy
gradient loss to enhance performance. Empirically, CompassJudger-2 achieves
superior results across multiple judge and reward benchmarks, and our 7B model
demonstrates competitive judgment accuracy with significantly larger models
like DeepSeek-V3 and Qwen3-235B-A22B. Additionally, we propose JudgerBenchV2, a
comprehensive benchmark evaluating cross-domain judgment accuracy and rank
consistency to standardize judge model evaluation. These contributions advance
robust, scalable LLM judgment and establish new performance and evaluation
standards.
6875bfaa257d4f0435370564
Taolin Zhang
6875bfaa257d4f0435370565
Maosong Cao
6875bfaa257d4f0435370566
Alexander Lam
claimed_verified
2025-07-21T08:30:30.752Z
63fc0d040aab060792003a68
/avatars/d573d4c4ec49513f3b157f7691becbdd.svg
Alexander Lam
user
acylam
6875bfaa257d4f0435370567
Songyang Zhang
claimed_verified
2025-07-16T08:15:54.280Z
630716d11801ecc7d2595021
/avatars/2d36a880ce4a3cf7efc5ff3987dbeaf3.svg
Songyang Zhang
user
zsytony
6875bfaa257d4f0435370568
Kai Chen
CompassJudger-2: Towards Generalist Judge Model via Verifiable Rewards
Recently, the role of LLM-as-judge in evaluating large language models has
gained prominence. However, current judge models suffer from narrow
specialization and limited robustness, undermining their capacity for
comprehensive evaluations. In this work, we present CompassJudger-2, a novel
generalist judge model that overcomes these limitations via a task-driven,
multi-domain data curation strategy. Central to our approach is supervising
judgment tasks with verifiable rewards, guiding intrinsic critical reasoning
through rejection sampling to foster robust, generalizable judgment
capabilities. We introduce a refined learning objective with margin policy
gradient loss to enhance performance. Empirically, CompassJudger-2 achieves
superior results across multiple judge and reward benchmarks, and our 7B model
demonstrates competitive judgment accuracy with significantly larger models
like DeepSeek-V3 and Qwen3-235B-A22B. Additionally, we propose JudgerBenchV2, a
comprehensive benchmark evaluating cross-domain judgment accuracy and rank
consistency to standardize judge model evaluation. These contributions advance
robust, scalable LLM judgment and establish new performance and evaluation
standards.
CompassJudger-2, a generalist judge model, improves cross-domain evaluation accuracy and robustness through task-driven data curation and a refined learning objective.
LLM-as-judge generalist judge model task-driven multi-domain data curation verifiable rewards rejection sampling margin policy gradient loss JudgerBenchV2 cross-domain judgment accuracy rank consistency