Why Do Open-Source LLMs Struggle with Data Analysis? A Systematic
  Empirical Study
Large Language Models (LLMs) hold promise in automating data analysis tasks,
yet open-source models face significant limitations in these kinds of
reasoning-intensive scenarios. In this work, we investigate strategies to
enhance the data analysis capabilities of open-source LLMs. By curating a seed
dataset of diverse, realistic scenarios, we evaluate models across three
dimensions: data understanding, code generation, and strategic planning. Our
analysis reveals three key findings: (1) Strategic planning quality serves as
the primary determinant of model performance; (2) Interaction design and task
complexity significantly influence reasoning capabilities; (3) Data quality
demonstrates a greater impact than diversity in achieving optimal performance.
We leverage these insights to develop a data synthesis methodology,
demonstrating significant improvements in open-source LLMs' analytical
reasoning capabilities.
685b75d0d2ee4fac76521e70
Yuqi Zhu
claimed_verified
2025-06-25T13:32:57.204Z
679b6d152583e6629337c592
/avatars/7ad77d7384fd41ad987b6eb5907a8f08.svg
Yuqi Zhu
user
Yukirsh
685b75d0d2ee4fac76521e71
Yi Zhong
685b75d0d2ee4fac76521e72
Jintian Zhang
685b75d0d2ee4fac76521e73
Ziheng Zhang
685b75d0d2ee4fac76521e74
Shuofei Qiao
685b75d0d2ee4fac76521e75
Yujie Luo
685b75d0d2ee4fac76521e76
Lun Du
685b75d0d2ee4fac76521e77
Da Zheng
685b75d0d2ee4fac76521e78
Huajun Chen
685b75d0d2ee4fac76521e79
Ningyu Zhang
claimed_verified
2025-06-25T11:04:12.367Z
620b3bbb0668e435407c8d0a
/avatars/e0fccbb2577d76088e09f054c35cffbc.svg
Ningyu Zhang
user
Ningyu
Why Do Open-Source LLMs Struggle with Data Analysis? A Systematic
  Empirical Study
Large Language Models (LLMs) hold promise in automating data analysis tasks,
yet open-source models face significant limitations in these kinds of
reasoning-intensive scenarios. In this work, we investigate strategies to
enhance the data analysis capabilities of open-source LLMs. By curating a seed
dataset of diverse, realistic scenarios, we evaluate models across three
dimensions: data understanding, code generation, and strategic planning. Our
analysis reveals three key findings: (1) Strategic planning quality serves as
the primary determinant of model performance; (2) Interaction design and task
complexity significantly influence reasoning capabilities; (3) Data quality
demonstrates a greater impact than diversity in achieving optimal performance.
We leverage these insights to develop a data synthesis methodology,
demonstrating significant improvements in open-source LLMs' analytical
reasoning capabilities.
Enhancements to open-source large language models' data analysis capabilities through strategic planning, interaction design, and data quality improvements were identified and applied.
Large Language Models data analysis data understanding code generation strategic planning interaction design task complexity data quality data synthesis methodology