INSTRUCTEVAL: Towards Holistic Evaluation of Instruction-Tuned Large
  Language Models
Instruction-tuned large language models have revolutionized natural language
processing and have shown great potential in applications such as
conversational agents. These models, such as GPT-4, can not only master
language but also solve complex tasks in areas like mathematics, coding,
medicine, and law. Despite their impressive capabilities, there is still a lack
of comprehensive understanding regarding their full potential, primarily due to
the black-box nature of many models and the absence of holistic evaluation
studies. To address these challenges, we present INSTRUCTEVAL, a more
comprehensive evaluation suite designed specifically for instruction-tuned
large language models. Unlike previous works, our evaluation involves a
rigorous assessment of models based on problem-solving, writing ability, and
alignment to human values. We take a holistic approach to analyze various
factors affecting model performance, including the pretraining foundation,
instruction-tuning data, and training methods. Our findings reveal that the
quality of instruction data is the most crucial factor in scaling model
performance. While open-source models demonstrate impressive writing abilities,
there is substantial room for improvement in problem-solving and alignment. We
are encouraged by the rapid development of models by the open-source community,
but we also highlight the need for rigorous evaluation to support claims made
about these models. Through INSTRUCTEVAL, we aim to foster a deeper
understanding of instruction-tuned models and advancements in their
capabilities. INSTRUCTEVAL is publicly available at
https://github.com/declare-lab/instruct-eval.
64827bfa120600dcbe532f44
Yew Ken Chia
64827bfa120600dcbe532f45
Pengfei Hong
admin_assigned
2023-06-09T16:22:39.632Z
62c3f44617c780a4f2ed603c
https://cdn-avatars.huggingface.co/v1/production/uploads/1657009213654-noauth.jpeg
Hong Pengfei
user
emrys-hong
64827bfa120600dcbe532f46
Lidong Bing
admin_assigned
2023-06-09T16:23:01.390Z
6454685a548f22be598414c4
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/eMjMWKJ-AouF7eY1-RzGF.jpeg
Lidong Bing
user
LidongBing
64827bfa120600dcbe532f47
Soujanya Poria
claimed_verified
2023-06-09T15:32:05.893Z
626b626405fe1cb65725aca1
https://cdn-avatars.huggingface.co/v1/production/uploads/626b626405fe1cb65725aca1/ZVSbhynzpQhVGq9kGywW6.png
Soujanya Poria
user
soujanyaporia
INSTRUCTEVAL: Towards Holistic Evaluation of Instruction-Tuned Large
  Language Models
Instruction-tuned large language models have revolutionized natural language
processing and have shown great potential in applications such as
conversational agents. These models, such as GPT-4, can not only master
language but also solve complex tasks in areas like mathematics, coding,
medicine, and law. Despite their impressive capabilities, there is still a lack
of comprehensive understanding regarding their full potential, primarily due to
the black-box nature of many models and the absence of holistic evaluation
studies. To address these challenges, we present INSTRUCTEVAL, a more
comprehensive evaluation suite designed specifically for instruction-tuned
large language models. Unlike previous works, our evaluation involves a
rigorous assessment of models based on problem-solving, writing ability, and
alignment to human values. We take a holistic approach to analyze various
factors affecting model performance, including the pretraining foundation,
instruction-tuning data, and training methods. Our findings reveal that the
quality of instruction data is the most crucial factor in scaling model
performance. While open-source models demonstrate impressive writing abilities,
there is substantial room for improvement in problem-solving and alignment. We
are encouraged by the rapid development of models by the open-source community,
but we also highlight the need for rigorous evaluation to support claims made
about these models. Through INSTRUCTEVAL, we aim to foster a deeper
understanding of instruction-tuned models and advancements in their
capabilities. INSTRUCTEVAL is publicly available at
https://github.com/declare-lab/instruct-eval.
INSTRUCTEVAL provides a comprehensive evaluation suite for instruction-tuned large language models, focusing on problem-solving, writing ability, and alignment to human values.