CompassJudger-1: All-in-one Judge Model Helps Model Evaluation and
  Evolution
Efficient and accurate evaluation is crucial for the continuous improvement
of large language models (LLMs). Among various assessment methods, subjective
evaluation has garnered significant attention due to its superior alignment
with real-world usage scenarios and human preferences. However, human-based
evaluations are costly and lack reproducibility, making precise automated
evaluators (judgers) vital in this process. In this report, we introduce
CompassJudger-1, the first open-source all-in-one judge LLM.
CompassJudger-1 is a general-purpose LLM that demonstrates remarkable
versatility. It is capable of: 1. Performing unitary scoring and two-model
comparisons as a reward model; 2. Conducting evaluations according to specified
formats; 3. Generating critiques; 4. Executing diverse tasks like a general
LLM. To assess the evaluation capabilities of different judge models under a
unified setting, we have also established JudgerBench, a new benchmark
that encompasses various subjective evaluation tasks and covers a wide range of
topics. CompassJudger-1 offers a comprehensive solution for various evaluation
tasks while maintaining the flexibility to adapt to diverse requirements. Both
CompassJudger and JudgerBench are released and available to the research
community athttps://github.com/open-compass/CompassJudger. We believe that by
open-sourcing these tools, we can foster collaboration and accelerate progress
in LLM evaluation methodologies.
671713d85d781c10ef4152b0
Maosong Cao
671713d85d781c10ef4152b1
Alexander Lam
admin_assigned
2024-10-22T08:37:21.537Z
63fc0d040aab060792003a68
/avatars/d573d4c4ec49513f3b157f7691becbdd.svg
Alexander Lam
user
acylam
671713d85d781c10ef4152b2
Haodong Duan
claimed_verified
2024-10-22T08:00:54.994Z
63ee1379190ddd6214efd73a
https://cdn-avatars.huggingface.co/v1/production/uploads/1676546883247-noauth.png
HAODONG DUAN
user
KennyUTC
671713d85d781c10ef4152b3
Hongwei Liu
671713d85d781c10ef4152b4
Songyang Zhang
claimed_verified
2024-10-22T08:00:56.898Z
630716d11801ecc7d2595021
/avatars/2d36a880ce4a3cf7efc5ff3987dbeaf3.svg
Songyang Zhang
user
zsytony
671713d85d781c10ef4152b5
Kai Chen
CompassJudger-1: All-in-one Judge Model Helps Model Evaluation and
  Evolution
Efficient and accurate evaluation is crucial for the continuous improvement
of large language models (LLMs). Among various assessment methods, subjective
evaluation has garnered significant attention due to its superior alignment
with real-world usage scenarios and human preferences. However, human-based
evaluations are costly and lack reproducibility, making precise automated
evaluators (judgers) vital in this process. In this report, we introduce
CompassJudger-1, the first open-source all-in-one judge LLM.
CompassJudger-1 is a general-purpose LLM that demonstrates remarkable
versatility. It is capable of: 1. Performing unitary scoring and two-model
comparisons as a reward model; 2. Conducting evaluations according to specified
formats; 3. Generating critiques; 4. Executing diverse tasks like a general
LLM. To assess the evaluation capabilities of different judge models under a
unified setting, we have also established JudgerBench, a new benchmark
that encompasses various subjective evaluation tasks and covers a wide range of
topics. CompassJudger-1 offers a comprehensive solution for various evaluation
tasks while maintaining the flexibility to adapt to diverse requirements. Both
CompassJudger and JudgerBench are released and available to the research
community athttps://github.com/open-compass/CompassJudger. We believe that by
open-sourcing these tools, we can foster collaboration and accelerate progress
in LLM evaluation methodologies.
CompassJudger-1 is an open-source, all-in-one judge LLM that evaluates LLMs through unitary scoring, model comparisons, critiques, and various tasks, supported by the JudgerBench benchmark.
LLMs subjective evaluation automated evaluators reward model evaluations according to specified formats generating critiques JudgerBench benchmark evaluation tasks