A Survey on Evaluation of Large Language Models
Large language models (LLMs) are gaining increasing popularity in both
academia and industry, owing to their unprecedented performance in various
applications. As LLMs continue to play a vital role in both research and daily
use, their evaluation becomes increasingly critical, not only at the task
level, but also at the society level for better understanding of their
potential risks. Over the past years, significant efforts have been made to
examine LLMs from various perspectives. This paper presents a comprehensive
review of these evaluation methods for LLMs, focusing on three key dimensions:
what to evaluate, where to evaluate, and how to evaluate. Firstly, we provide
an overview from the perspective of evaluation tasks, encompassing general
natural language processing tasks, reasoning, medical usage, ethics,
educations, natural and social sciences, agent applications, and other areas.
Secondly, we answer the `where' and `how' questions by diving into the
evaluation methods and benchmarks, which serve as crucial components in
assessing performance of LLMs. Then, we summarize the success and failure cases
of LLMs in different tasks. Finally, we shed light on several future challenges
that lie ahead in LLMs evaluation. Our aim is to offer invaluable insights to
researchers in the realm of LLMs evaluation, thereby aiding the development of
more proficient LLMs. Our key point is that evaluation should be treated as an
essential discipline to better assist the development of LLMs. We consistently
maintain the related open-source materials at:
https://github.com/MLGroupJLU/LLM-eval-survey.
64a787b2fc1d1ebc8a84867f
Yupeng Chang
64a787b2fc1d1ebc8a848680
Xu Wang
64a787b2fc1d1ebc8a848681
Jindong Wang
claimed_verified
2023-07-10T09:24:21.647Z
6204cc0d522e40b4a18d86e2
/avatars/18daf2de5671e711dc745388dd60569d.svg
Jindong Wang
user
jindongwang
64a787b2fc1d1ebc8a848682
Yuan Wu
claimed_verified
2024-10-17T11:31:34.945Z
670e57b3391f1a7021182bff
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/N0tuHZVz8KFPCv8G1qUX2.png
Yuan Wu
user
WhiteCatY
64a787b2fc1d1ebc8a848683
Kaijie Zhu
claimed_verified
2024-05-06T07:06:00.081Z
641fb3733c49fc33cedb4429
/avatars/3577bca726282ed4b26009f3a2d5e87d.svg
Kaijie Zhu
user
March07
64a787b2fc1d1ebc8a848684
Hao Chen
64a787b2fc1d1ebc8a848685
Linyi Yang
claimed_verified
2023-07-07T13:31:06.181Z
64895683f534abe18eec264b
/avatars/73cc9e6db6db86793787750776b57c63.svg
Linyi Yang
user
linyiyang2023
64a787b2fc1d1ebc8a848686
Xiaoyuan Yi
64a787b2fc1d1ebc8a848687
Cunxiang Wang
64a787b2fc1d1ebc8a848688
Yidong Wang
64a787b2fc1d1ebc8a848689
Wei Ye
64a787b2fc1d1ebc8a84868a
Yue Zhang
64a787b2fc1d1ebc8a84868b
Yi Chang
64a787b2fc1d1ebc8a84868c
Philip S. Yu
64a787b2fc1d1ebc8a84868d
Qiang Yang
64a787b2fc1d1ebc8a84868e
Xing Xie
A Survey on Evaluation of Large Language Models
Large language models (LLMs) are gaining increasing popularity in both
academia and industry, owing to their unprecedented performance in various
applications. As LLMs continue to play a vital role in both research and daily
use, their evaluation becomes increasingly critical, not only at the task
level, but also at the society level for better understanding of their
potential risks. Over the past years, significant efforts have been made to
examine LLMs from various perspectives. This paper presents a comprehensive
review of these evaluation methods for LLMs, focusing on three key dimensions:
what to evaluate, where to evaluate, and how to evaluate. Firstly, we provide
an overview from the perspective of evaluation tasks, encompassing general
natural language processing tasks, reasoning, medical usage, ethics,
educations, natural and social sciences, agent applications, and other areas.
Secondly, we answer the `where' and `how' questions by diving into the
evaluation methods and benchmarks, which serve as crucial components in
assessing performance of LLMs. Then, we summarize the success and failure cases
of LLMs in different tasks. Finally, we shed light on several future challenges
that lie ahead in LLMs evaluation. Our aim is to offer invaluable insights to
researchers in the realm of LLMs evaluation, thereby aiding the development of
more proficient LLMs. Our key point is that evaluation should be treated as an
essential discipline to better assist the development of LLMs. We consistently
maintain the related open-source materials at:
https://github.com/MLGroupJLU/LLM-eval-survey.
A comprehensive review of evaluation methods for large language models (LLMs) focusing on tasks, methods, and future challenges.
large language models LLMs natural language processing reasoning ethics education natural sciences social sciences agent applications evaluation methods benchmarks task evaluation performance assessment