Language Complexity Measurement as a Noisy Zero-Shot Proxy for
  Evaluating LLM Performance
Large Language Models (LLMs) have made significant strides in natural
language generation but often face challenges in tasks requiring precise
calculations and structural analysis. This paper investigates the performance
of state-of-the-art LLMs on language complexity measurement tasks, through the
computation of the LIX readability metric and Average Dependency Distance
(ADD). Using Swedish high school and university-level essays, we evaluate the
models' abilities to compute LIX scores and perform dependency parsing,
comparing their results to established ground truths. Our findings reveal that
while all models demonstrate some capacity for these tasks, ChatGPT-o1-mini
performs most consistently, achieving the highest accuracy in both LIX
computation and dependency parsing. Additionally, we observe a strong
significant correlation -0.875 p 0.026 (N=6) between the models' accuracy in
computing LIX and their overall performance on the Massive Multitask Language
Understanding (MMLU) benchmark. These results suggest that language complexity
measurement abilities can serve as a noisy zero-shot proxies for assessing the
general capabilities of LLMs, providing a practical method for model evaluation
without the need for extensive benchmarking datasets.
67b435475bff5f34c1ebee1b
Birger Moell
claimed_verified
2025-02-18T09:30:52.639Z
6033e34a9aa44495c80dd043
https://cdn-avatars.huggingface.co/v1/production/uploads/1614079701740-6033e34a9aa44495c80dd043.jpeg
Birger Moell
user
birgermoell
67b435475bff5f34c1ebee1c
Johan Boye
extracted_pending
2025-02-18T07:22:48.554Z
65bcbc01d6d0ffbceb8b2e6e
/avatars/73edb2d6b7b11208439ac88b365079e8.svg
Johan Boye
user
jboye
Language Complexity Measurement as a Noisy Zero-Shot Proxy for
  Evaluating LLM Performance
Large Language Models (LLMs) have made significant strides in natural
language generation but often face challenges in tasks requiring precise
calculations and structural analysis. This paper investigates the performance
of state-of-the-art LLMs on language complexity measurement tasks, through the
computation of the LIX readability metric and Average Dependency Distance
(ADD). Using Swedish high school and university-level essays, we evaluate the
models' abilities to compute LIX scores and perform dependency parsing,
comparing their results to established ground truths. Our findings reveal that
while all models demonstrate some capacity for these tasks, ChatGPT-o1-mini
performs most consistently, achieving the highest accuracy in both LIX
computation and dependency parsing. Additionally, we observe a strong
significant correlation -0.875 p 0.026 (N=6) between the models' accuracy in
computing LIX and their overall performance on the Massive Multitask Language
Understanding (MMLU) benchmark. These results suggest that language complexity
measurement abilities can serve as a noisy zero-shot proxies for assessing the
general capabilities of LLMs, providing a practical method for model evaluation
without the need for extensive benchmarking datasets.
Evaluation of state-of-the-art LLMs on language complexity measurement tasks shows ChatGPT-o1-mini performs most consistently, and accuracy in LIX computation correlates significantly with MMLU benchmark performance.
Large Language Models LIX readability metric Average Dependency Distance dependency parsing Massive Multitask Language Understanding (MMLU) benchmark