Calibrating LLM-Based Evaluator
Recent advancements in large language models (LLMs) on language modeling and
emergent capabilities make them a promising reference-free evaluator of natural
language generation quality, and a competent alternative to human evaluation.
However, hindered by the closed-source or high computational demand to host and
tune, there is a lack of practice to further calibrate an off-the-shelf
LLM-based evaluator towards better human alignment. In this work, we propose
AutoCalibrate, a multi-stage, gradient-free approach to automatically calibrate
and align an LLM-based evaluator toward human preference. Instead of explicitly
modeling human preferences, we first implicitly encompass them within a set of
human labels. Then, an initial set of scoring criteria is drafted by the
language model itself, leveraging in-context learning on different few-shot
examples. To further calibrate this set of criteria, we select the best
performers and re-draft them with self-refinement. Our experiments on multiple
text quality evaluation datasets illustrate a significant improvement in
correlation with expert evaluation through calibration. Our comprehensive
qualitative analysis conveys insightful intuitions and observations on the
essence of effective scoring criteria.
651271be2f0aa026dd6733df
Yuxuan Liu
651271be2f0aa026dd6733e0
Tianchi Yang
admin_assigned
2023-09-26T13:07:33.805Z
63b640f20d5913eee48bff48
/avatars/a3ab337cdccaaac0652dba60ce581b14.svg
yangtianchi
user
asinica
651271be2f0aa026dd6733e1
Shaohan Huang
admin_assigned
2023-09-26T13:07:47.142Z
632bd2f72d6a805eeb4bc601
/avatars/6e1533e8a599f3068290aa69ac82cab7.svg
HUANG SHAOHAN
user
buaahsh
651271be2f0aa026dd6733e2
Zihan Zhang
651271be2f0aa026dd6733e3
Haizhen Huang
claimed_verified
2023-10-23T09:16:23.761Z
6302e8e57e137e3d6b3f5a06
/avatars/aa82fe4fb1212dc120ecccd406568c5d.svg
Huang
user
Haizhen
651271be2f0aa026dd6733e4
Furu Wei
admin_assigned
2023-09-26T13:21:39.779Z
6368c512fbfe97c16a40baba
/avatars/1c23bc7c0b6d9225699ce27647623d7a.svg
Furu Wei
user
thegenerality
651271be2f0aa026dd6733e5
Weiwei Deng
admin_assigned
2023-09-26T13:23:56.509Z
642da49dc1adada5b345c72b
/avatars/5fa8d345a7248246454444999756105b.svg
deng
user
weiweideng
651271be2f0aa026dd6733e6
Feng Sun
admin_assigned
2023-09-26T13:24:04.058Z
64b88a7449bde5d9481e4755
/avatars/45a963517ec2c54c81ee80449d655014.svg
feng sun
user
forserty
651271be2f0aa026dd6733e7
Qi Zhang
Calibrating LLM-Based Evaluator
Recent advancements in large language models (LLMs) on language modeling and
emergent capabilities make them a promising reference-free evaluator of natural
language generation quality, and a competent alternative to human evaluation.
However, hindered by the closed-source or high computational demand to host and
tune, there is a lack of practice to further calibrate an off-the-shelf
LLM-based evaluator towards better human alignment. In this work, we propose
AutoCalibrate, a multi-stage, gradient-free approach to automatically calibrate
and align an LLM-based evaluator toward human preference. Instead of explicitly
modeling human preferences, we first implicitly encompass them within a set of
human labels. Then, an initial set of scoring criteria is drafted by the
language model itself, leveraging in-context learning on different few-shot
examples. To further calibrate this set of criteria, we select the best
performers and re-draft them with self-refinement. Our experiments on multiple
text quality evaluation datasets illustrate a significant improvement in
correlation with expert evaluation through calibration. Our comprehensive
qualitative analysis conveys insightful intuitions and observations on the
essence of effective scoring criteria.
AutoCalibrate uses an iterative, gradient-free approach to automatically align an off-the-shelf LLM-based evaluator with human preferences, improving its correlation with expert evaluations.
large language models LLMs language modeling emergent capabilities reference-free evaluator natural language generation human evaluation closed-source computational demand human labels in-context learning few-shot examples self-refinement text quality evaluation expert evaluation scoring criteria