Cross-Lingual Auto Evaluation for Assessing Multilingual LLMs
Evaluating machine-generated text remains a significant challenge in NLP,
especially for non-English languages. Current methodologies, including
automated metrics, human assessments, and LLM-based evaluations, predominantly
focus on English, revealing a significant gap in multilingual evaluation
frameworks. We introduce the Cross Lingual Auto Evaluation (CIA) Suite, an
extensible framework that includes evaluator LLMs (Hercule) and a novel test
set (Recon) specifically designed for multilingual evaluation. Our test set
features 500 human-annotated instructions spanning various task capabilities
along with human judgment scores across six languages. This would enable
benchmarking of general-purpose multilingual LLMs and facilitate
meta-evaluation of Evaluator LLMs. The proposed model, Hercule, is a
cross-lingual evaluation model that addresses the scarcity of reference answers
in the target language by learning to assign scores to responses based on
easily available reference answers in English. Our experiments demonstrate that
Hercule aligns more closely with human judgments compared to proprietary
models, demonstrating the effectiveness of such cross-lingual evaluation in low
resource scenarios. Further, it is also effective in zero-shot evaluation on
unseen languages. This study is the first comprehensive examination of
cross-lingual evaluation using LLMs, presenting a scalable and effective
approach for multilingual assessment. All code, datasets, and models will be
publicly available to enable further research in this important area.
67121b74e82a3853eda3b069
Sumanth Doddapaneni
admin_assigned
2024-10-22T09:48:58.988Z
5fbfc7a6e366524fe8e97ccf
https://cdn-avatars.huggingface.co/v1/production/uploads/1673518581924-5fbfc7a6e366524fe8e97ccf.png
Sumanth Doddapaneni
user
sumanthd
67121b74e82a3853eda3b06a
Mohammed Safi Ur Rahman Khan
claimed_verified
2024-10-18T12:15:16.098Z
63ef3cd11e695b35aa48bebc
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/BnOgZiv1IHKbtTaFTWpAZ.png
Mohammed Safi Ur Rahman Khan
user
safikhan
67121b74e82a3853eda3b06b
Dilip Venkatesh
admin_assigned
2024-10-22T09:50:07.904Z
6408278e8dca6cec91cb7e55
/avatars/fc599425785202727b1a21b89af552bb.svg
Dilip Venkatesh
user
dipsivenkatesh
67121b74e82a3853eda3b06c
Raj Dabre
admin_assigned
2024-10-22T09:50:02.568Z
61cd4af60996e2a209ff7b96
/avatars/7b47bfaba15dfaf37ce38cb0216f3102.svg
Raj Dabre
user
prajdabre
67121b74e82a3853eda3b06d
Anoop Kunchukuttan
admin_assigned
2024-10-22T09:49:57.100Z
5fce179fb3dbf216ad31837b
https://cdn-avatars.huggingface.co/v1/production/uploads/1659894079257-5fce179fb3dbf216ad31837b.jpeg
Anoop Kunchukuttan
user
anoopk
67121b74e82a3853eda3b06e
Mitesh M. Khapra
admin_assigned
2024-10-22T09:49:50.912Z
6551c181c0e106160e9d3297
/avatars/d2c498d7f361f4133d1104fde72c24b9.svg
Mitesh Khapra
user
miteshk
Cross-Lingual Auto Evaluation for Assessing Multilingual LLMs
Evaluating machine-generated text remains a significant challenge in NLP,
especially for non-English languages. Current methodologies, including
automated metrics, human assessments, and LLM-based evaluations, predominantly
focus on English, revealing a significant gap in multilingual evaluation
frameworks. We introduce the Cross Lingual Auto Evaluation (CIA) Suite, an
extensible framework that includes evaluator LLMs (Hercule) and a novel test
set (Recon) specifically designed for multilingual evaluation. Our test set
features 500 human-annotated instructions spanning various task capabilities
along with human judgment scores across six languages. This would enable
benchmarking of general-purpose multilingual LLMs and facilitate
meta-evaluation of Evaluator LLMs. The proposed model, Hercule, is a
cross-lingual evaluation model that addresses the scarcity of reference answers
in the target language by learning to assign scores to responses based on
easily available reference answers in English. Our experiments demonstrate that
Hercule aligns more closely with human judgments compared to proprietary
models, demonstrating the effectiveness of such cross-lingual evaluation in low
resource scenarios. Further, it is also effective in zero-shot evaluation on
unseen languages. This study is the first comprehensive examination of
cross-lingual evaluation using LLMs, presenting a scalable and effective
approach for multilingual assessment. All code, datasets, and models will be
publicly available to enable further research in this important area.
The Cross Lingual Auto Evaluation (CIA) Suite introduces a framework with evaluator LLMs and a multilingual test set for benchmarking general-purpose multilingual LLMs and meta-evaluating evaluator models.
Cross Lingual Auto Evaluation CIA Suite evaluator LLMs Hercule multilingual evaluation human-annotated instructions zero-shot evaluation low resource scenarios