CodeMMLU: A Multi-Task Benchmark for Assessing Code Understanding
  Capabilities of CodeLLMs
Recent advancements in Code Large Language Models (CodeLLMs) have
predominantly focused on open-ended code generation tasks, often neglecting the
critical aspect of code understanding and comprehension. To bridge this gap, we
present CodeMMLU, a comprehensive multiple-choice question-answer benchmark
designed to evaluate the depth of software and code understanding in LLMs.
CodeMMLU includes over 10,000 questions sourced from diverse domains,
encompassing tasks such as code analysis, defect detection, and software
engineering principles across multiple programming languages. Unlike
traditional benchmarks, CodeMMLU assesses models's ability to reason about code
rather than merely generate it, providing deeper insights into their grasp of
complex software concepts and systems. Our extensive evaluation reveals that
even state-of-the-art models face significant challenges with CodeMMLU,
highlighting deficiencies in comprehension beyond code generation. By
underscoring the crucial relationship between code understanding and effective
generation, CodeMMLU serves as a vital resource for advancing AI-assisted
software development, ultimately aiming to create more reliable and capable
coding assistants.
66ffb6ae03aaa32983158167
Dung Nguyen Manh
claimed_verified
2024-10-15T09:11:29.107Z
63733f7fd398fce0dd45125c
https://cdn-avatars.huggingface.co/v1/production/uploads/63733f7fd398fce0dd45125c/VzqcN9YCjMDekopqyDS-y.png
Nguyen Manh Dung
user
nmd2k
66ffb6ae03aaa32983158168
Thang Phan Chau
66ffb6ae03aaa32983158169
Nam Le Hai
claimed_verified
2024-10-08T08:17:32.517Z
63a27e21f30c4642278de3ea
https://cdn-avatars.huggingface.co/v1/production/uploads/63a27e21f30c4642278de3ea/zj0mGDv9DEK9JYHFjfaZq.jpeg
Le Hai Nam
user
NamCyan
66ffb6ae03aaa3298315816a
Thong T. Doan
claimed_verified
2024-10-08T08:17:28.637Z
63e8ad2accae1fe5c61878d6
https://cdn-avatars.huggingface.co/v1/production/uploads/1676193023976-noauth.jpeg
Doan Tien Thong
user
doantienthongbku
66ffb6ae03aaa3298315816b
Nam V. Nguyen
claimed_verified
2024-10-08T08:17:30.357Z
64c2bea2ada7df214276913b
https://cdn-avatars.huggingface.co/v1/production/uploads/64c2bea2ada7df214276913b/QFCtmCn439Afsr7uqyoMT.jpeg
Nguyen Van Nam
user
DavidNguyen
66ffb6ae03aaa3298315816c
Quang Pham
66ffb6ae03aaa3298315816d
Nghi D. Q. Bui
CodeMMLU: A Multi-Task Benchmark for Assessing Code Understanding
  Capabilities of CodeLLMs
Recent advancements in Code Large Language Models (CodeLLMs) have
predominantly focused on open-ended code generation tasks, often neglecting the
critical aspect of code understanding and comprehension. To bridge this gap, we
present CodeMMLU, a comprehensive multiple-choice question-answer benchmark
designed to evaluate the depth of software and code understanding in LLMs.
CodeMMLU includes over 10,000 questions sourced from diverse domains,
encompassing tasks such as code analysis, defect detection, and software
engineering principles across multiple programming languages. Unlike
traditional benchmarks, CodeMMLU assesses models's ability to reason about code
rather than merely generate it, providing deeper insights into their grasp of
complex software concepts and systems. Our extensive evaluation reveals that
even state-of-the-art models face significant challenges with CodeMMLU,
highlighting deficiencies in comprehension beyond code generation. By
underscoring the crucial relationship between code understanding and effective
generation, CodeMMLU serves as a vital resource for advancing AI-assisted
software development, ultimately aiming to create more reliable and capable
coding assistants.
CodeMMLU evaluates code understanding in LLMs through a comprehensive multiple-choice benchmark, revealing significant gaps in comprehension compared to open-ended code generation.
Code Large Language Models CodeMMLU multiple-choice question-answer benchmark code analysis defect detection software engineering principles programming languages reasoning about code AI-assisted software development coding assistants