CodeArena: A Collective Evaluation Platform for LLM Code Generation
Large Language Models (LLMs) have reshaped code generation by synergizing
their exceptional comprehension of natural language and programming syntax,
thereby substantially boosting developer productivity. These advancements have
prompted numerous efforts to quantitatively evaluate their coding capabilities.
However, persistent challenges, such as benchmark leakage, data dissipation,
and limited system accessibility, continue to impede a timely and accurate
assessment. To address these limitations, we introduce CodeArena, an online
evaluation framework tailored for LLM code generation. The key innovation is a
collective evaluation mechanism, which dynamically recalibrates individual
model scores based on the holistic performance of all participating models,
mitigating score biases caused by widespread benchmark leakage. In addition,
CodeArena ensures open access to all submitted solutions and test cases and
provides automation-friendly APIs to streamline the code evaluation workflow.
Our main contributions are: (1) a collective evaluation system for unbiased
assessment, (2) a public repository of solutions and test cases, and (3)
automation-ready APIs for seamless integration.
67c6a8b534aeb86063e94010
Mingzhe Du
claimed_verified
2025-03-04T08:34:49.954Z
61711f02e0b1ddb56eb9b526
/avatars/3e2fdf774f5bc1f73b450486d6da42d4.svg
Mingzhe Du
user
Elfsong
67c6a8b534aeb86063e94011
Anh Tuan Luu
admin_assigned
2025-03-04T11:02:20.575Z
655722e80438e0854fae7554
/avatars/b93a74f7c7880f9fe0f3ffb47e2aef5e.svg
Luu Anh Tuan
user
anhtuanluu36
67c6a8b534aeb86063e94012
Bin Ji
67c6a8b534aeb86063e94013
Xiaobao Wu
admin_assigned
2025-03-04T11:02:48.996Z
64cb02869e30a46f7b80b355
/avatars/81ce4ba78826b54f0e1b53eeaff87ee6.svg
Xiaobao Wu
user
bobxwu
67c6a8b534aeb86063e94014
Dong Huang
admin_assigned
2025-03-04T11:02:43.013Z
67c56a7f083bb2c50254bbe5
/avatars/bdf6fd8934c2199ff169b178f6482773.svg
Huang, Dong
user
DongHuang-ebay
67c6a8b534aeb86063e94015
Terry Yue Zhuo
admin_assigned
2025-03-04T11:02:33.977Z
62b7fb545233925f253531c8
https://cdn-avatars.huggingface.co/v1/production/uploads/62b7fb545233925f253531c8/W50u2G1HK3EtUKHRU189V.jpeg
Terry Yue Zhuo
user
terryyz
67c6a8b534aeb86063e94016
Qian Liu
67c6a8b534aeb86063e94017
See-Kiong Ng
CodeArena: A Collective Evaluation Platform for LLM Code Generation
Large Language Models (LLMs) have reshaped code generation by synergizing
their exceptional comprehension of natural language and programming syntax,
thereby substantially boosting developer productivity. These advancements have
prompted numerous efforts to quantitatively evaluate their coding capabilities.
However, persistent challenges, such as benchmark leakage, data dissipation,
and limited system accessibility, continue to impede a timely and accurate
assessment. To address these limitations, we introduce CodeArena, an online
evaluation framework tailored for LLM code generation. The key innovation is a
collective evaluation mechanism, which dynamically recalibrates individual
model scores based on the holistic performance of all participating models,
mitigating score biases caused by widespread benchmark leakage. In addition,
CodeArena ensures open access to all submitted solutions and test cases and
provides automation-friendly APIs to streamline the code evaluation workflow.
Our main contributions are: (1) a collective evaluation system for unbiased
assessment, (2) a public repository of solutions and test cases, and (3)
automation-ready APIs for seamless integration.
CodeArena is an online framework that provides unbiased code generation evaluation for Large Language Models through collective recalibration and accessible resources.
Large Language Models code generation natural language programming syntax benchmark leakage data dissipation system accessibility online evaluation framework collective evaluation mechanism public repository automation-friendly APIs