RevisEval: Improving LLM-as-a-Judge via Response-Adapted References
With significant efforts in recent studies, LLM-as-a-Judge has become a
cost-effective alternative to human evaluation for assessing the text
generation quality in a wide range of tasks. However, there still remains a
reliability gap between LLM-as-a-Judge and human evaluation. One important
reason is the lack of guided oracles in the evaluation process. Motivated by
the role of reference pervasively used in classic text evaluation, we introduce
RevisEval, a novel text generation evaluation paradigm via the response-adapted
references. RevisEval is driven by the key observation that an ideal reference
should maintain the necessary relevance to the response to be evaluated.
Specifically, RevisEval leverages the text revision capabilities of large
language models (LLMs) to adaptively revise the response, then treat the
revised text as the reference (response-adapted reference) for the subsequent
evaluation. Extensive experiments demonstrate that RevisEval outperforms
traditional reference-free and reference-based evaluation paradigms that use
LLM-as-a-Judge across NLG tasks and open-ended instruction-following tasks.
More importantly, our response-adapted references can further boost the
classical text metrics, e.g., BLEU and BERTScore, compared to traditional
references and even rival the LLM-as-a-Judge. A detailed analysis is also
conducted to confirm RevisEval's effectiveness in bias reduction, the impact of
inference cost, and reference relevance.
6704e69412e582928046c613
Qiyuan Zhang
claimed_verified
2024-10-08T15:16:43.381Z
62a42f22c683d02f5b63320c
/avatars/bc611abe9c4ef8d378123cb8ac9fdbf2.svg
Qiyuan Zhang
user
DonJoey
6704e69412e582928046c614
Yufei Wang
admin_assigned
2024-10-09T09:17:57.837Z
667c1c0b18aae25ed39b96c1
/avatars/3513e7ae5d2cc8767e09893605123ee2.svg
Yufei Wang
user
yufeiwang201217a
6704e69412e582928046c615
Tiezheng YU
admin_assigned
2024-10-09T09:18:04.918Z
61bd88aa94bd578f7286181a
/avatars/0f40dfd7accc558f9201103e3e4ec007.svg
Tiezheng Yu
user
Tiezheng
6704e69412e582928046c616
Yuxin Jiang
admin_assigned
2024-10-09T09:18:38.205Z
63c20105726f62e411fbe882
https://cdn-avatars.huggingface.co/v1/production/uploads/63c20105726f62e411fbe882/2UsU9O2psbDjJzz-sAmGH.jpeg
Yuxin Jiang
user
YuxinJiang
6704e69412e582928046c617
Chuhan Wu
6704e69412e582928046c618
Liangyou Li
6704e69412e582928046c619
Yasheng Wang
6704e69412e582928046c61a
Xin Jiang
6704e69412e582928046c61b
Lifeng Shang
admin_assigned
2024-10-09T09:20:33.974Z
655b1360c11dee7f7e7cf794
/avatars/efb4b91e9bb8ab531331c8e4296f754c.svg
lifengshang
user
lifengshang
6704e69412e582928046c61c
Ruiming Tang
6704e69412e582928046c61d
Fuyuan Lyu
claimed_verified
2024-10-09T12:54:01.875Z
65d2bb5c6130ef7be012d235
/avatars/1c1e3bbb2c683a5c9d1f792a2c13fc4a.svg
Fuyuan Lyu
user
silentspring2
6704e69412e582928046c61e
Chen Ma
RevisEval: Improving LLM-as-a-Judge via Response-Adapted References
With significant efforts in recent studies, LLM-as-a-Judge has become a
cost-effective alternative to human evaluation for assessing the text
generation quality in a wide range of tasks. However, there still remains a
reliability gap between LLM-as-a-Judge and human evaluation. One important
reason is the lack of guided oracles in the evaluation process. Motivated by
the role of reference pervasively used in classic text evaluation, we introduce
RevisEval, a novel text generation evaluation paradigm via the response-adapted
references. RevisEval is driven by the key observation that an ideal reference
should maintain the necessary relevance to the response to be evaluated.
Specifically, RevisEval leverages the text revision capabilities of large
language models (LLMs) to adaptively revise the response, then treat the
revised text as the reference (response-adapted reference) for the subsequent
evaluation. Extensive experiments demonstrate that RevisEval outperforms
traditional reference-free and reference-based evaluation paradigms that use
LLM-as-a-Judge across NLG tasks and open-ended instruction-following tasks.
More importantly, our response-adapted references can further boost the
classical text metrics, e.g., BLEU and BERTScore, compared to traditional
references and even rival the LLM-as-a-Judge. A detailed analysis is also
conducted to confirm RevisEval's effectiveness in bias reduction, the impact of
inference cost, and reference relevance.
RevisEval, a novel evaluation paradigm using response-adapted references generated by LLMs, improves text generation assessments over traditional methods.
LLM-as-a-Judge text generation evaluation response-adapted references large language models reference-free reference-based evaluation NLG tasks bias reduction inference cost BLEU BERTScore