CS-Bench: A Comprehensive Benchmark for Large Language Models towards
  Computer Science Mastery
Computer Science (CS) stands as a testament to the intricacies of human
intelligence, profoundly advancing the development of artificial intelligence
and modern society. However, the current community of large language models
(LLMs) overly focuses on benchmarks for analyzing specific foundational skills
(e.g. mathematics and code generation), neglecting an all-round evaluation of
the computer science field. To bridge this gap, we introduce CS-Bench, the
first bilingual (Chinese-English) benchmark dedicated to evaluating the
performance of LLMs in computer science. CS-Bench comprises approximately 5K
meticulously curated test samples, covering 26 subfields across 4 key areas of
computer science, encompassing various task forms and divisions of knowledge
and reasoning. Utilizing CS-Bench, we conduct a comprehensive evaluation of
over 30 mainstream LLMs, revealing the relationship between CS performance and
model scales. We also quantitatively analyze the reasons for failures in
existing LLMs and highlight directions for improvements, including knowledge
supplementation and CS-specific reasoning. Further cross-capability experiments
show a high correlation between LLMs' capabilities in computer science and
their abilities in mathematics and coding. Moreover, expert LLMs specialized in
mathematics and coding also demonstrate strong performances in several CS
subfields. Looking ahead, we envision CS-Bench serving as a cornerstone for LLM
applications in the CS field and paving new avenues in assessing LLMs' diverse
reasoning capabilities. The CS-Bench data and evaluation code are available at
https://github.com/csbench/csbench.
666bb1e1fba6c458fcf39d70
Xiaoshuai Song
666bb1e1fba6c458fcf39d71
Muxi Diao
claimed_verified
2024-12-30T19:40:27.978Z
6768c97367e4b4606a3c9cec
/avatars/5ddafe7a05828366f66c79072556f370.svg
diaomuxi
user
diaomuxi
666bb1e1fba6c458fcf39d72
Guanting Dong
claimed_verified
2024-06-30T20:58:07.854Z
61cd4b833dd34ba1985e0753
https://cdn-avatars.huggingface.co/v1/production/uploads/61cd4b833dd34ba1985e0753/BfHfrwotoMESpXZOHiIe4.png
KABI
user
dongguanting
666bb1e1fba6c458fcf39d73
Zhengyang Wang
666bb1e1fba6c458fcf39d74
Yujia Fu
666bb1e1fba6c458fcf39d75
Runqi Qiao
claimed_verified
2024-12-02T08:52:23.974Z
6683a05e74fb1736a4b7c934
https://cdn-avatars.huggingface.co/v1/production/uploads/6683a05e74fb1736a4b7c934/eiz6qlqIUjAWGy5zfg8Cs.jpeg
QRQ
user
RichardQRQ
666bb1e1fba6c458fcf39d76
Zhexu Wang
claimed_verified
2025-07-11T13:13:30.764Z
666aab1be0ecb2edac62e914
/avatars/8eda456501bff02c4f625a568c7f2350.svg
yugongwzx
user
yugongwzx
666bb1e1fba6c458fcf39d77
Dayuan Fu
admin_assigned
2024-06-14T12:01:07.517Z
65572fd2bcdc315c0a1395d7
/avatars/36135ca35bbd4baca37a9092f3e2d82c.svg
Fu-Dayuan
user
fudayuan
666bb1e1fba6c458fcf39d78
Huangxuan Wu
666bb1e1fba6c458fcf39d79
Bin Liang
666bb1e1fba6c458fcf39d7a
Weihao Zeng
claimed_verified
2024-07-18T09:08:33.995Z
62751082b43ccfeef483424f
/avatars/fec83e4478e7d1731ba6033328131852.svg
WeihaoZeng
user
AndrewZeng
666bb1e1fba6c458fcf39d7b
Yejie Wang
claimed_verified
2024-09-09T10:58:14.074Z
6342dc0ee2647466b42918ab
/avatars/a80a3df2f67410662bf9681ed8834b17.svg
Yejie Wang
user
banksy235
666bb1e1fba6c458fcf39d7c
Zhuoma GongQue
claimed_verified
2024-12-20T08:34:33.573Z
6762b973686e574cab4717ec
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/icacS5LkbX2S46b3Dqbni.png
zhuoma
user
Ethereal1l
666bb1e1fba6c458fcf39d7d
Jianing Yu
666bb1e1fba6c458fcf39d7e
Qiuna Tan
666bb1e1fba6c458fcf39d7f
Weiran Xu
CS-Bench: A Comprehensive Benchmark for Large Language Models towards
  Computer Science Mastery
Computer Science (CS) stands as a testament to the intricacies of human
intelligence, profoundly advancing the development of artificial intelligence
and modern society. However, the current community of large language models
(LLMs) overly focuses on benchmarks for analyzing specific foundational skills
(e.g. mathematics and code generation), neglecting an all-round evaluation of
the computer science field. To bridge this gap, we introduce CS-Bench, the
first bilingual (Chinese-English) benchmark dedicated to evaluating the
performance of LLMs in computer science. CS-Bench comprises approximately 5K
meticulously curated test samples, covering 26 subfields across 4 key areas of
computer science, encompassing various task forms and divisions of knowledge
and reasoning. Utilizing CS-Bench, we conduct a comprehensive evaluation of
over 30 mainstream LLMs, revealing the relationship between CS performance and
model scales. We also quantitatively analyze the reasons for failures in
existing LLMs and highlight directions for improvements, including knowledge
supplementation and CS-specific reasoning. Further cross-capability experiments
show a high correlation between LLMs' capabilities in computer science and
their abilities in mathematics and coding. Moreover, expert LLMs specialized in
mathematics and coding also demonstrate strong performances in several CS
subfields. Looking ahead, we envision CS-Bench serving as a cornerstone for LLM
applications in the CS field and paving new avenues in assessing LLMs' diverse
reasoning capabilities. The CS-Bench data and evaluation code are available at
https://github.com/csbench/csbench.
CS-Bench evaluates large language models across various subfields of computer science, uncovering performance trends and pinpointing areas for improvement.
large language models LLMs CS-Bench bilingual benchmark computer science test samples subfields knowledge supplementation CS-specific reasoning diverse reasoning capabilities