Judging with Confidence: Calibrating Autoraters to Preference
  Distributions
The alignment of large language models (LLMs) with human values increasingly
relies on using other LLMs as automated judges, or ``autoraters''. However,
their reliability is limited by a foundational issue: they are trained on
discrete preference labels, forcing a single ground truth onto tasks that are
often subjective, ambiguous, or nuanced. We argue that a reliable autorater
must learn to model the full distribution of preferences defined by a target
population. In this paper, we propose a general framework for calibrating
probabilistic autoraters to any given preference distribution. We formalize the
problem and present two learning methods tailored to different data conditions:
1) a direct supervised fine-tuning for dense, probabilistic labels, and 2) a
reinforcement learning approach for sparse, binary labels. Our empirical
results show that finetuning autoraters with a distribution-matching objective
leads to verbalized probability predictions that are better aligned with the
target preference distribution, with improved calibration and significantly
lower positional bias, all while preserving performance on objective tasks.
68e44581e4e093a7044e4c86
Zhuohang Li
claimed_verified
2025-10-07T12:27:29.972Z
6328f05f84e4c8b42c5d14a4
/avatars/f8277830c7fc4978f0fc0535b25177cd.svg
Zhuohang Li
user
zhhli
68e44581e4e093a7044e4c87
Xiaowei Li
68e44581e4e093a7044e4c88
Chengyu Huang
68e44581e4e093a7044e4c89
Guowang Li
68e44581e4e093a7044e4c8a
Katayoon Goshvadi
68e44581e4e093a7044e4c8b
Bo Dai
68e44581e4e093a7044e4c8c
Dale Schuurmans
68e44581e4e093a7044e4c8d
Paul Zhou
68e44581e4e093a7044e4c8e
Hamid Palangi
68e44581e4e093a7044e4c8f
Yiwen Song
68e44581e4e093a7044e4c90
Palash Goyal
68e44581e4e093a7044e4c91
Murat Kantarcioglu
68e44581e4e093a7044e4c92
Bradley A. Malin
68e44581e4e093a7044e4c93
Yuan Xue
Judging with Confidence: Calibrating Autoraters to Preference
  Distributions
The alignment of large language models (LLMs) with human values increasingly
relies on using other LLMs as automated judges, or ``autoraters''. However,
their reliability is limited by a foundational issue: they are trained on
discrete preference labels, forcing a single ground truth onto tasks that are
often subjective, ambiguous, or nuanced. We argue that a reliable autorater
must learn to model the full distribution of preferences defined by a target
population. In this paper, we propose a general framework for calibrating
probabilistic autoraters to any given preference distribution. We formalize the
problem and present two learning methods tailored to different data conditions:
1) a direct supervised fine-tuning for dense, probabilistic labels, and 2) a
reinforcement learning approach for sparse, binary labels. Our empirical
results show that finetuning autoraters with a distribution-matching objective
leads to verbalized probability predictions that are better aligned with the
target preference distribution, with improved calibration and significantly
lower positional bias, all while preserving performance on objective tasks.
A framework for calibrating probabilistic autoraters to preference distributions using supervised fine-tuning and reinforcement learning improves alignment with human values and reduces bias.
large language models LLMs autoraters preference labels probabilistic autoraters distribution-matching objective verbalized probability predictions calibration positional bias