On Robustness and Reliability of Benchmark-Based Evaluation of LLMs
Large Language Models (LLMs) effectiveness is usually evaluated by means of
benchmarks such as MMLU, ARC-C, or HellaSwag, where questions are presented in
their original wording, thus in a fixed, standardized format. However,
real-world applications involve linguistic variability, requiring models to
maintain their effectiveness across diverse rewordings of the same question or
query. In this study, we systematically assess the robustness of LLMs to
paraphrased benchmark questions and investigate whether benchmark-based
evaluations provide a reliable measure of model capabilities. We systematically
generate various paraphrases of all the questions across six different common
benchmarks, and measure the resulting variations in effectiveness of 34
state-of-the-art LLMs, of different size and effectiveness. Our findings reveal
that while LLM rankings remain relatively stable across paraphrased inputs,
absolute effectiveness scores change, and decline significantly. This suggests
that LLMs struggle with linguistic variability, raising concerns about their
generalization abilities and evaluation methodologies. Furthermore, the
observed performance drop challenges the reliability of benchmark-based
evaluations, indicating that high benchmark scores may not fully capture a
model's robustness to real-world input variations. We discuss the implications
of these findings for LLM evaluation methodologies, emphasizing the need for
robustness-aware benchmarks that better reflect practical deployment scenarios.
68be6ac4c123124955ef60cb
Riccardo Lunardi
68be6ac4c123124955ef60cc
Vincenzo Della Mea
68be6ac4c123124955ef60cd
Stefano Mizzaro
68be6ac4c123124955ef60ce
Kevin Roitero
On Robustness and Reliability of Benchmark-Based Evaluation of LLMs
Large Language Models (LLMs) effectiveness is usually evaluated by means of
benchmarks such as MMLU, ARC-C, or HellaSwag, where questions are presented in
their original wording, thus in a fixed, standardized format. However,
real-world applications involve linguistic variability, requiring models to
maintain their effectiveness across diverse rewordings of the same question or
query. In this study, we systematically assess the robustness of LLMs to
paraphrased benchmark questions and investigate whether benchmark-based
evaluations provide a reliable measure of model capabilities. We systematically
generate various paraphrases of all the questions across six different common
benchmarks, and measure the resulting variations in effectiveness of 34
state-of-the-art LLMs, of different size and effectiveness. Our findings reveal
that while LLM rankings remain relatively stable across paraphrased inputs,
absolute effectiveness scores change, and decline significantly. This suggests
that LLMs struggle with linguistic variability, raising concerns about their
generalization abilities and evaluation methodologies. Furthermore, the
observed performance drop challenges the reliability of benchmark-based
evaluations, indicating that high benchmark scores may not fully capture a
model's robustness to real-world input variations. We discuss the implications
of these findings for LLM evaluation methodologies, emphasizing the need for
robustness-aware benchmarks that better reflect practical deployment scenarios.
LLMs show reduced effectiveness on paraphrased benchmark questions, indicating limitations in handling linguistic variability and suggesting the need for more robust evaluation methods.
Large Language Models LLMs MMLU ARC-C HellaSwag paraphrased benchmark questions linguistic variability generalization abilities benchmark-based evaluations robustness-aware benchmarks