Annotation-Efficient Universal Honesty Alignment
Honesty alignment-the ability of large language models (LLMs) to recognize
their knowledge boundaries and express calibrated confidence-is essential for
trustworthy deployment. Existing methods either rely on training-free
confidence estimation (e.g., token probabilities, self-consistency) or
training-based calibration with correctness annotations. While effective,
achieving universal honesty alignment with training-based calibration requires
costly, large-scale labeling. To support annotation-efficient training, we
introduce Elicitation-Then-Calibration (EliCal), a two-stage framework that
first elicits internal confidence using inexpensive self-consistency
supervision, then calibrates this confidence with a small set of correctness
annotations. To support a large-scale study, we release HonestyBench, a
benchmark covering ten free-form QA datasets with 560k training and 70k
evaluation instances annotated with correctness and self-consistency signals.
Experiments show that EliCal achieves near-optimal alignment with only 1k
correctness annotations (0.18% of full supervision) and better alignment
performance on unseen MMLU tasks than the calibration-only baseline, offering a
scalable solution toward universal honesty alignment in LLMs.
68f704d324c4489363111991
Shiyu Ni
claimed_verified
2025-10-21T14:03:08.864Z
64b75440da8017900e76a073
/avatars/28b4096bf13c4fdcd560015a79cd8332.svg
Shiyu Ni
user
Shiyunee
68f704d324c4489363111992
Keping Bi
68f704d324c4489363111993
Jiafeng Guo
68f704d324c4489363111994
Minghao Tang
claimed_verified
2025-10-21T14:03:05.642Z
66544459258f5fad314c56bb
/avatars/283212e48ff267ecb19a5d1ab1ffd29d.svg
Minghao Tang
user
StudentTang
68f704d324c4489363111995
Jingtong Wu
68f704d324c4489363111996
Zengxin Han
68f704d324c4489363111997
Xueqi Cheng
Annotation-Efficient Universal Honesty Alignment
Honesty alignment-the ability of large language models (LLMs) to recognize
their knowledge boundaries and express calibrated confidence-is essential for
trustworthy deployment. Existing methods either rely on training-free
confidence estimation (e.g., token probabilities, self-consistency) or
training-based calibration with correctness annotations. While effective,
achieving universal honesty alignment with training-based calibration requires
costly, large-scale labeling. To support annotation-efficient training, we
introduce Elicitation-Then-Calibration (EliCal), a two-stage framework that
first elicits internal confidence using inexpensive self-consistency
supervision, then calibrates this confidence with a small set of correctness
annotations. To support a large-scale study, we release HonestyBench, a
benchmark covering ten free-form QA datasets with 560k training and 70k
evaluation instances annotated with correctness and self-consistency signals.
Experiments show that EliCal achieves near-optimal alignment with only 1k
correctness annotations (0.18% of full supervision) and better alignment
performance on unseen MMLU tasks than the calibration-only baseline, offering a
scalable solution toward universal honesty alignment in LLMs.
EliCal, a two-stage framework combining self-consistency supervision and minimal correctness annotations, achieves near-optimal honesty alignment in large language models with limited annotation effort.
large language models honesty alignment confidence estimation self-consistency calibration correctness annotations Elicitation-Then-Calibration EliCal HonestyBench free-form QA datasets MMLU tasks