AutoCodeBench: Large Language Models are Automatic Code Benchmark
  Generators
Large Language Models (LLMs) have demonstrated remarkable capabilities across
various domains, with code generation emerging as a key area of focus. While
numerous benchmarks have been proposed to evaluate their code generation
abilities, these benchmarks face several critical limitations. First, they
often rely on manual annotations, which are time-consuming and difficult to
scale across different programming languages and problem complexities. Second,
most existing benchmarks focus primarily on Python, while the few multilingual
benchmarks suffer from limited difficulty and uneven language distribution. To
address these challenges, we propose AutoCodeGen, an automated method for
generating high-difficulty multilingual code generation datasets without manual
annotations. AutoCodeGen ensures the correctness and completeness of test cases
by generating test inputs with LLMs and obtaining test outputs through a
multilingual sandbox, while achieving high data quality through reverse-order
problem generation and multiple filtering steps. Using this novel method, we
introduce AutoCodeBench, a large-scale code generation benchmark comprising
3,920 problems evenly distributed across 20 programming languages. It is
specifically designed to evaluate LLMs on challenging, diverse, and practical
multilingual tasks. We evaluate over 30 leading open-source and proprietary
LLMs on AutoCodeBench and its simplified version AutoCodeBench-Lite. The
results show that even the most advanced LLMs struggle with the complexity,
diversity, and multilingual nature of these tasks. Besides, we introduce
AutoCodeBench-Complete, specifically designed for base models to assess their
few-shot code generation capabilities. We hope the AutoCodeBench series will
serve as a valuable resource and inspire the community to focus on more
challenging and practical multilingual code generation scenarios.
689c00c7fab6fdd2e52ac84b
Jason Chou
689c00c7fab6fdd2e52ac84c
Ao Liu
689c00c7fab6fdd2e52ac84d
Yuchi Deng
689c00c7fab6fdd2e52ac84e
Zhiying Zeng
689c00c7fab6fdd2e52ac84f
Tao Zhang
689c00c7fab6fdd2e52ac850
Haotian Zhu
689c00c7fab6fdd2e52ac851
Jianwei Cai
689c00c7fab6fdd2e52ac852
Yue Mao
689c00c7fab6fdd2e52ac853
Chenchen Zhang
689c00c7fab6fdd2e52ac854
Lingyun Tan
689c00c7fab6fdd2e52ac855
Ziyan Xu
689c00c7fab6fdd2e52ac856
Bohui Zhai
689c00c7fab6fdd2e52ac857
Hengyi Liu
689c00c7fab6fdd2e52ac858
Speed Zhu
689c00c7fab6fdd2e52ac859
Wiggin Zhou
689c00c7fab6fdd2e52ac85a
Fengzong Lian
AutoCodeBench: Large Language Models are Automatic Code Benchmark
  Generators
Large Language Models (LLMs) have demonstrated remarkable capabilities across
various domains, with code generation emerging as a key area of focus. While
numerous benchmarks have been proposed to evaluate their code generation
abilities, these benchmarks face several critical limitations. First, they
often rely on manual annotations, which are time-consuming and difficult to
scale across different programming languages and problem complexities. Second,
most existing benchmarks focus primarily on Python, while the few multilingual
benchmarks suffer from limited difficulty and uneven language distribution. To
address these challenges, we propose AutoCodeGen, an automated method for
generating high-difficulty multilingual code generation datasets without manual
annotations. AutoCodeGen ensures the correctness and completeness of test cases
by generating test inputs with LLMs and obtaining test outputs through a
multilingual sandbox, while achieving high data quality through reverse-order
problem generation and multiple filtering steps. Using this novel method, we
introduce AutoCodeBench, a large-scale code generation benchmark comprising
3,920 problems evenly distributed across 20 programming languages. It is
specifically designed to evaluate LLMs on challenging, diverse, and practical
multilingual tasks. We evaluate over 30 leading open-source and proprietary
LLMs on AutoCodeBench and its simplified version AutoCodeBench-Lite. The
results show that even the most advanced LLMs struggle with the complexity,
diversity, and multilingual nature of these tasks. Besides, we introduce
AutoCodeBench-Complete, specifically designed for base models to assess their
few-shot code generation capabilities. We hope the AutoCodeBench series will
serve as a valuable resource and inspire the community to focus on more
challenging and practical multilingual code generation scenarios.
AutoCodeGen creates a large-scale, multilingual code generation benchmark, AutoCodeBench, to evaluate LLMs on diverse and complex tasks without manual annotations.
Large Language Models code generation benchmarks manual annotations multilingual test cases multilingual sandbox reverse-order problem generation filtering steps AutoCodeBench few-shot code generation AutoCodeBench-Lite AutoCodeBench-Complete