Are LLMs Better than Reported? Detecting Label Errors and Mitigating
  Their Effect on Model Performance
NLP benchmarks rely on standardized datasets for training and evaluating
models and are crucial for advancing the field. Traditionally, expert
annotations ensure high-quality labels; however, the cost of expert annotation
does not scale well with the growing demand for larger datasets required by
modern models. While crowd-sourcing provides a more scalable solution, it often
comes at the expense of annotation precision and consistency. Recent
advancements in large language models (LLMs) offer new opportunities to enhance
the annotation process, particularly for detecting label errors in existing
datasets. In this work, we consider the recent approach of LLM-as-a-judge,
leveraging an ensemble of LLMs to flag potentially mislabeled examples. Through
a case study of four datasets from the TRUE benchmark, covering different tasks
and domains, we empirically analyze the labeling quality of existing datasets,
and compare expert, crowd-sourced, and our LLM-based annotations in terms of
agreement, label quality, and efficiency, demonstrating the strengths and
limitations of each annotation method. Our findings reveal a substantial number
of label errors, which, when corrected, induce a significant upward shift in
reported model performance. This suggests that many of the LLMs so-called
mistakes are due to label errors rather than genuine model failures.
Additionally, we discuss the implications of mislabeled data and propose
methods to mitigate them in training to improve model performance.
671f8971915aa3d2d1691b9b
Omer Nahum
admin_assigned
2024-10-28T15:37:31.730Z
663511a0efbee9a1fde43940
/avatars/ad124ac100829359f67052c730e0a0d4.svg
Omer Nahum
user
omer6nahum
671f8971915aa3d2d1691b9c
Nitay Calderon
admin_assigned
2024-10-28T15:37:37.337Z
62d6a0c18faee0ac953c51fa
/avatars/ca818cebdb089a8d853c5bc4d5e0987b.svg
Nitay Calderon
user
nitay
671f8971915aa3d2d1691b9d
Orgad Keller
admin_assigned
2024-10-28T15:37:42.433Z
66f00fe216bffdccd51832e2
/avatars/6d2330d2353ed40cbff6bd4a71fb7865.svg
Orgad Keller
user
orgad
671f8971915aa3d2d1691b9e
Idan Szpektor
671f8971915aa3d2d1691b9f
Roi Reichart
Are LLMs Better than Reported? Detecting Label Errors and Mitigating
  Their Effect on Model Performance
NLP benchmarks rely on standardized datasets for training and evaluating
models and are crucial for advancing the field. Traditionally, expert
annotations ensure high-quality labels; however, the cost of expert annotation
does not scale well with the growing demand for larger datasets required by
modern models. While crowd-sourcing provides a more scalable solution, it often
comes at the expense of annotation precision and consistency. Recent
advancements in large language models (LLMs) offer new opportunities to enhance
the annotation process, particularly for detecting label errors in existing
datasets. In this work, we consider the recent approach of LLM-as-a-judge,
leveraging an ensemble of LLMs to flag potentially mislabeled examples. Through
a case study of four datasets from the TRUE benchmark, covering different tasks
and domains, we empirically analyze the labeling quality of existing datasets,
and compare expert, crowd-sourced, and our LLM-based annotations in terms of
agreement, label quality, and efficiency, demonstrating the strengths and
limitations of each annotation method. Our findings reveal a substantial number
of label errors, which, when corrected, induce a significant upward shift in
reported model performance. This suggests that many of the LLMs so-called
mistakes are due to label errors rather than genuine model failures.
Additionally, we discuss the implications of mislabeled data and propose
methods to mitigate them in training to improve model performance.
LLMs used as judges can identify label errors in datasets, improving annotation quality and model performance across various tasks and domains.
large language models LLM-as-a-judge ensemble models TRUE benchmark labeling quality annotation efficiency