Are We Done with MMLU?
Maybe not. We identify and analyse errors in the popular Massive Multitask
Language Understanding (MMLU) benchmark. Even though MMLU is widely adopted,
our analysis demonstrates numerous ground truth errors that obscure the true
capabilities of LLMs. For example, we find that 57% of the analysed questions
in the Virology subset contain errors. To address this issue, we introduce a
comprehensive framework for identifying dataset errors using a novel error
taxonomy. Then, we create MMLU-Redux, which is a subset of 3,000 manually
re-annotated questions across 30 MMLU subjects. Using MMLU-Redux, we
demonstrate significant discrepancies with the model performance metrics that
were originally reported. Our results strongly advocate for revising MMLU's
error-ridden questions to enhance its future utility and reliability as a
benchmark. Therefore, we open up MMLU-Redux for additional annotation
https://huggingface.co/datasets/edinburgh-dawg/mmlu-redux.
66636e5e0b562c29c5bdda45
Aryo Pradipta Gema
admin_assigned
2024-06-13T19:54:02.123Z
644f895e23d7eb05ca695054
/avatars/3fb04dd8544b403262bf98507de05453.svg
Aryo Pradipta Gema
user
aryopg
66636e5e0b562c29c5bdda46
Joshua Ong Jun Leang
admin_assigned
2024-06-13T19:54:08.747Z
6537dd20aa214745cbc89ad9
/avatars/cb6bda58ae6063562eb02cf92c0e7c4e.svg
Joshua Ong Jun Leang
user
Jforeverss
66636e5e0b562c29c5bdda47
Giwon Hong
admin_assigned
2024-06-13T19:54:14.275Z
653c1a31afb09482c62df54f
/avatars/e11b00995fd32e421019018a90359eb0.svg
Giwon Hong
user
GWHed
66636e5e0b562c29c5bdda48
Alessio Devoto
admin_assigned
2024-06-13T19:54:20.984Z
62180288d05d07ebf7de5f31
https://cdn-avatars.huggingface.co/v1/production/uploads/62180288d05d07ebf7de5f31/io2RsaRSjIDktQylXVvEB.jpeg
Alessio
user
alessiodevoto
66636e5e0b562c29c5bdda49
Alberto Carlo Maria Mancino
admin_assigned
2024-06-13T19:54:27.659Z
663a42b49b8151660f4c4cba
/avatars/f53b2da495386df834fbc7e4c35eee1d.svg
Alberto Carlo Maria Mancino
user
albertomancino
66636e5e0b562c29c5bdda4a
Rohit Saxena
claimed_verified
2024-06-13T13:46:33.918Z
657ccbf2869d5bb0e53b482f
/avatars/2eae5a10bdc14814a04d9f255f16de6b.svg
Rohit Saxena
user
rohitsaxena
66636e5e0b562c29c5bdda4b
Xuanli He
admin_assigned
2024-06-13T19:54:33.526Z
614b0ab851b6682b9e56fd92
/avatars/81606ae85106b89a34bd08283646d07b.svg
Xuanli He
user
zodiache
66636e5e0b562c29c5bdda4c
Yu Zhao
claimed_verified
2024-06-09T20:04:38.695Z
654a500d9b8bd6406d431c0d
/avatars/a6b76441bbc6f4b71d49c52e454c9ef7.svg
Yu Zhao
user
yuzhaouoe
66636e5e0b562c29c5bdda4d
Xiaotang Du
claimed_verified
2024-11-15T09:46:56.843Z
63d4698d1da8c28b7b8ab04a
/avatars/73d4f49018db03a59e72b69bb2d1dd0c.svg
Xiaotang Du
user
acDante
66636e5e0b562c29c5bdda4e
Mohammad Reza Ghasemi Madani
claimed_verified
2024-06-13T12:00:24.500Z
63d03e196b985b0f25c91bcc
/avatars/974fa4ed6f368602436a305377365152.svg
Mohammad Reza Ghasemi Madani
user
Reza-Madani
66636e5e0b562c29c5bdda4f
Claire Barale
admin_assigned
2024-06-13T19:54:39.624Z
652d6dd40517d2c3746fb73a
/avatars/9c8b5167b98a33a5f85aa1a6eb50eb36.svg
Claire Barale
user
clairebarale
66636e5e0b562c29c5bdda50
Robert McHardy
admin_assigned
2024-06-13T19:54:57.806Z
64b91a3fa62c52b252cf39b2
/avatars/af80d43464d8a8bacdb32c5b77d1b001.svg
Robert McHardy 
user
robmchinst
66636e5e0b562c29c5bdda51
Joshua Harris
claimed_verified
2025-05-12T09:04:01.587Z
64cb98c6f103036e23c69b1d
/avatars/7ee33880ad39f5335b618dc53554124a.svg
Harris
user
Joshua-Harris
66636e5e0b562c29c5bdda52
Jean Kaddour
admin_assigned
2024-06-13T19:55:17.616Z
6304061c0547362a22a76a17
https://cdn-avatars.huggingface.co/v1/production/uploads/1661339692442-6304061c0547362a22a76a17.jpeg
Jean Kaddour
user
JeanKaddour
66636e5e0b562c29c5bdda53
Emile van Krieken
claimed_verified
2024-06-13T12:00:23.118Z
62cd4ae83e5ba89c40f15156
/avatars/f70aca85f8836edaedcee324a18140b5.svg
Emile van Krieken
user
HEmile
66636e5e0b562c29c5bdda54
Pasquale Minervini
claimed_verified
2024-06-13T12:00:21.688Z
61001311e043e15c13412d30
https://cdn-avatars.huggingface.co/v1/production/uploads/61001311e043e15c13412d30/6yAbTweYR16XtxMBEyOWl.png
Pasquale Minervini
user
pminervini
Are We Done with MMLU?
Maybe not. We identify and analyse errors in the popular Massive Multitask
Language Understanding (MMLU) benchmark. Even though MMLU is widely adopted,
our analysis demonstrates numerous ground truth errors that obscure the true
capabilities of LLMs. For example, we find that 57% of the analysed questions
in the Virology subset contain errors. To address this issue, we introduce a
comprehensive framework for identifying dataset errors using a novel error
taxonomy. Then, we create MMLU-Redux, which is a subset of 3,000 manually
re-annotated questions across 30 MMLU subjects. Using MMLU-Redux, we
demonstrate significant discrepancies with the model performance metrics that
were originally reported. Our results strongly advocate for revising MMLU's
error-ridden questions to enhance its future utility and reliability as a
benchmark. Therefore, we open up MMLU-Redux for additional annotation
https://huggingface.co/datasets/edinburgh-dawg/mmlu-redux.
Analysis identifies numerous errors in the MMLU benchmark, leading to the creation of MMLU-Redux, a re-annotated subset that highlights discrepancies in model performance metrics.
Massive Multitask Language Understanding MMLU ground truth errors LLMs error taxonomy dataset errors MMLU-Redux model performance metrics