FLASK: Fine-grained Language Model Evaluation based on Alignment Skill
  Sets
Evaluation of Large Language Models (LLMs) is challenging because aligning to
human values requires the composition of multiple skills and the required set
of skills varies depending on the instruction. Recent studies have evaluated
the performance of LLMs in two ways, (1) automatic evaluation on several
independent benchmarks and (2) human or machined-based evaluation giving an
overall score to the response. However, both settings are coarse-grained
evaluations, not considering the nature of user instructions that require
instance-wise skill composition, which limits the interpretation of the true
capabilities of LLMs. In this paper, we introduce FLASK (Fine-grained Language
Model Evaluation based on Alignment SKill Sets), a fine-grained evaluation
protocol that can be used for both model-based and human-based evaluation which
decomposes coarse-level scoring to an instance-wise skill set-level.
Specifically, we define 12 fine-grained skills needed for LLMs to follow
open-ended user instructions and construct an evaluation set by allocating a
set of skills for each instance. Additionally, by annotating the target domains
and difficulty level for each instance, FLASK provides a holistic view with a
comprehensive analysis of a model's performance depending on skill, domain, and
difficulty. Through using FLASK, we compare multiple open-sourced and
proprietary LLMs and observe highly-correlated findings between model-based and
human-based evaluations. FLASK enables developers to more accurately measure
the model performance and how it can be improved by analyzing factors that make
LLMs proficient in particular skills. For practitioners, FLASK can be used to
recommend suitable models for particular situations through comprehensive
comparison among various LLMs. We release the evaluation data and code
implementation at https://github.com/kaistAI/FLASK.
64ba0fc1b3bd26c44b114b8d
Seonghyeon Ye
admin_assigned
2023-07-21T07:59:37.015Z
62551f7767f0b85962624047
https://cdn-avatars.huggingface.co/v1/production/uploads/1664552038624-62551f7767f0b85962624047.png
Seonghyeon Ye
user
seonghyeonye
64ba0fc1b3bd26c44b114b8e
Doyoung Kim
admin_assigned
2023-07-21T07:59:59.393Z
610659fc377b3983fe49b3a7
https://cdn-avatars.huggingface.co/v1/production/uploads/1665487693252-610659fc377b3983fe49b3a7.jpeg
doyoung kim
user
doyoungkim
64ba0fc1b3bd26c44b114b8f
Sungdong Kim
admin_assigned
2023-07-21T08:00:18.381Z
6084ebe956f7623a0b849153
/avatars/c56fb056800c145e5a8234e27b6b5a31.svg
Sungdong kim
user
dsksd
64ba0fc1b3bd26c44b114b90
Hyeonbin Hwang
admin_assigned
2023-07-21T08:00:32.770Z
647eaaf61a1fcad2fdc5d1ef
/avatars/596d7a61d6e6227d38b661210a32fed0.svg
Hyeonbin Hwang 
user
hbin0701
64ba0fc1b3bd26c44b114b91
Seungone Kim
admin_assigned
2023-07-21T08:01:11.845Z
6469949654873f0043b09c22
https://cdn-avatars.huggingface.co/v1/production/uploads/6469949654873f0043b09c22/Lk7IJAR16Wa_sGJ2g81AQ.jpeg
Seungone Kim
user
seungone
64ba0fc1b3bd26c44b114b92
Yongrae Jo
admin_assigned
2023-07-21T08:01:26.466Z
649e06313e5e7504763dfe03
/avatars/1c4d19de5f2950d3342480c4b3e01047.svg
Yongrae Jo
user
dreamgonfly
64ba0fc1b3bd26c44b114b93
James Thorne
admin_assigned
2023-07-21T08:01:59.704Z
63b54db5889aa6707f08f2d1
https://cdn-avatars.huggingface.co/v1/production/uploads/63b54db5889aa6707f08f2d1/zUnyx-8Zabxr54poZtUo4.jpeg
James Thorne
user
j6mes
64ba0fc1b3bd26c44b114b94
Juho Kim
admin_assigned
2023-07-21T08:02:24.631Z
62d6411d4d766e27a4736782
/avatars/16f259d27134beafdd28620046bdfcc6.svg
Juho Kim
user
mcpanic
64ba0fc1b3bd26c44b114b95
Minjoon Seo
admin_assigned
2023-07-21T08:02:43.071Z
621f05ba970615ad5861ceb1
/avatars/7e1902aa71369a524afda9b0a9e88e22.svg
Minjoon Seo
user
minjoon
FLASK: Fine-grained Language Model Evaluation based on Alignment Skill
  Sets
Evaluation of Large Language Models (LLMs) is challenging because aligning to
human values requires the composition of multiple skills and the required set
of skills varies depending on the instruction. Recent studies have evaluated
the performance of LLMs in two ways, (1) automatic evaluation on several
independent benchmarks and (2) human or machined-based evaluation giving an
overall score to the response. However, both settings are coarse-grained
evaluations, not considering the nature of user instructions that require
instance-wise skill composition, which limits the interpretation of the true
capabilities of LLMs. In this paper, we introduce FLASK (Fine-grained Language
Model Evaluation based on Alignment SKill Sets), a fine-grained evaluation
protocol that can be used for both model-based and human-based evaluation which
decomposes coarse-level scoring to an instance-wise skill set-level.
Specifically, we define 12 fine-grained skills needed for LLMs to follow
open-ended user instructions and construct an evaluation set by allocating a
set of skills for each instance. Additionally, by annotating the target domains
and difficulty level for each instance, FLASK provides a holistic view with a
comprehensive analysis of a model's performance depending on skill, domain, and
difficulty. Through using FLASK, we compare multiple open-sourced and
proprietary LLMs and observe highly-correlated findings between model-based and
human-based evaluations. FLASK enables developers to more accurately measure
the model performance and how it can be improved by analyzing factors that make
LLMs proficient in particular skills. For practitioners, FLASK can be used to
recommend suitable models for particular situations through comprehensive
comparison among various LLMs. We release the evaluation data and code
implementation at https://github.com/kaistAI/FLASK.
FLASK evaluates large language models using a fine-grained protocol that assesses skills required for following user instructions across different domains and difficulty levels.
large language models FLASK fine-grained evaluation skill composition user instructions model evaluation domain-specific difficulty levels model-based evaluation human-based evaluation