BigCodeArena: Unveiling More Reliable Human Preferences in Code
  Generation via Execution
Crowdsourced model evaluation platforms, such as Chatbot Arena, enable
real-time evaluation from human perspectives to assess the quality of model
responses. In the coding domain, manually examining the quality of
LLM-generated content is extremely challenging, as it requires understanding
long chunks of raw code and deliberately simulating code execution. To this
end, we introduce BigCodeArena, an open human evaluation platform for code
generation backed by a comprehensive and on-the-fly execution environment.
Built on top of Chatbot Arena, BigCodeArena enables the execution of
LLM-generated code and allows humans to interact with the execution process and
outcomes. We collected over 14,000 raw code-centric conversation sessions
across 10 widely used LLMs, spanning 10 languages and 8 types of execution
environments. Among these conversations, we identified more than 4,700
multi-turn samples with pairwise human preferences. Further analysis uncovers
underexplored preferences of LLMs in fine-grained domains characterized by
tasks, languages, and frameworks. To systematically examine code understanding
and generation capabilities of frontier LLMs, we curated two benchmarks based
on the collected data, namely BigCodeReward and AutoCodeArena. For
BigCodeReward, we post-processed the 4,700 conversations and evaluated the
consistency between reward models and human preferences. The evaluation shows
that most LLMs have superior performance in judging coding preferences when the
execution results are available. Inspired by these findings, we propose
AutoCodeArena, an automatic Elo rating benchmark designed to assess the coding
quality of LLMs without human involvement. We find that proprietary LLMs like
GPT-5, Claude-Sonnet-4, and Claude-Opus-4 still lead in code generation
performance among recent emerging models.
68ec5e33cd07fb414898c90f
Terry Yue Zhuo
claimed_verified
2025-10-14T07:32:32.606Z
62b7fb545233925f253531c8
https://cdn-avatars.huggingface.co/v1/production/uploads/62b7fb545233925f253531c8/W50u2G1HK3EtUKHRU189V.jpeg
Terry Yue Zhuo
user
terryyz
68ec5e33cd07fb414898c910
Xiaolong Jin
68ec5e33cd07fb414898c911
Hange Liu
claimed_verified
2025-10-14T07:32:28.359Z
68ed502c3af6e7780c27e225
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/5sOHq7dCHekpKQcPN8IUR.png
Hange Liu
user
hngl
68ec5e33cd07fb414898c912
Juyong Jiang
68ec5e33cd07fb414898c913
Tianyang Liu
68ec5e33cd07fb414898c914
Chen Gong
68ec5e33cd07fb414898c915
Bhupesh Bishnoi
68ec5e33cd07fb414898c916
Vaisakhi Mishra
68ec5e33cd07fb414898c917
Marek Suppa
68ec5e33cd07fb414898c918
Noah Ziems
68ec5e33cd07fb414898c919
Saiteja Utpala
68ec5e33cd07fb414898c91a
Ming Xu
68ec5e33cd07fb414898c91b
Guangyu Song
68ec5e33cd07fb414898c91c
Kaixin Li
claimed_verified
2025-10-14T07:32:21.682Z
6346be8f7fb9f11870c63998
https://cdn-avatars.huggingface.co/v1/production/uploads/6346be8f7fb9f11870c63998/tFWawSkXL6bv1zgvzFWQd.png
Kaixin Li
user
likaixin
68ec5e33cd07fb414898c91d
Yuhan Cao
68ec5e33cd07fb414898c91e
Bo Liu
claimed_verified
2025-10-13T10:06:03.836Z
635e3a76106f984574c36409
https://cdn-avatars.huggingface.co/v1/production/uploads/1667120725800-635e3a76106f984574c36409.png
Bo Liu
user
Benjamin-eecs
68ec5e33cd07fb414898c91f
Zheng Liu
68ec5e33cd07fb414898c920
Sabina Abdurakhmanova
68ec5e33cd07fb414898c921
Wenhao Yu
68ec5e33cd07fb414898c922
Mengzhao Jia
68ec5e33cd07fb414898c923
Jihan Yao
68ec5e33cd07fb414898c924
Kenneth Hamilton
claimed_verified
2025-10-14T07:32:30.438Z
656e3808d4de03a07d116850
https://cdn-avatars.huggingface.co/v1/production/uploads/656e3808d4de03a07d116850/62cFw46AmuhdI3gS24F1M.jpeg
Kenneth Hamilton
user
ZennyKenny
68ec5e33cd07fb414898c925
Kumar Shridhar
68ec5e33cd07fb414898c926
Minh Chien Vu
claimed_verified
2025-10-14T07:32:24.000Z
60535c9d10aba34e3b6a2ef7
https://cdn-avatars.huggingface.co/v1/production/uploads/1616075850230-noauth.jpeg
vumichien
user
vumichien
68ec5e33cd07fb414898c927
Dingmin Wang
68ec5e33cd07fb414898c928
Jiawei Liu
68ec5e33cd07fb414898c929
Zijian Wang
68ec5e33cd07fb414898c92a
Qian Liu
68ec5e33cd07fb414898c92b
Binyuan Hui
68ec5e33cd07fb414898c92c
Meg Risdal
68ec5e33cd07fb414898c92d
Ahsen Khaliq
admin_assigned
2025-10-13T12:48:46.417Z
60f1abe7544c2adfd699860c
https://cdn-avatars.huggingface.co/v1/production/uploads/1674929746905-60f1abe7544c2adfd699860c.jpeg
AK
user
akhaliq
68ec5e33cd07fb414898c92e
Atin Sood
68ec5e33cd07fb414898c92f
Zhenchang Xing
68ec5e33cd07fb414898c930
Wasi Uddin Ahmad
68ec5e33cd07fb414898c931
John Grundy
68ec5e33cd07fb414898c932
David Lo
68ec5e33cd07fb414898c933
Banghua Zhu
68ec5e33cd07fb414898c934
Xiaoning Du
68ec5e33cd07fb414898c935
Torsten Scholak
claimed_verified
2025-10-17T04:14:25.480Z
60ecaa5efee13fee7ada7af4
https://cdn-avatars.huggingface.co/v1/production/uploads/1637694419284-60ecaa5efee13fee7ada7af4.jpeg
Torsten Scholak
user
tscholak
68ec5e33cd07fb414898c936
Leandro von Werra
claimed_verified
2025-10-14T07:32:26.258Z
5e48005437cb5b49818287a5
https://cdn-avatars.huggingface.co/v1/production/uploads/5e48005437cb5b49818287a5/4uCXGGui-9QifAT4qelxU.png
Leandro von Werra
user
lvwerra
BigCodeArena: Unveiling More Reliable Human Preferences in Code
  Generation via Execution
Crowdsourced model evaluation platforms, such as Chatbot Arena, enable
real-time evaluation from human perspectives to assess the quality of model
responses. In the coding domain, manually examining the quality of
LLM-generated content is extremely challenging, as it requires understanding
long chunks of raw code and deliberately simulating code execution. To this
end, we introduce BigCodeArena, an open human evaluation platform for code
generation backed by a comprehensive and on-the-fly execution environment.
Built on top of Chatbot Arena, BigCodeArena enables the execution of
LLM-generated code and allows humans to interact with the execution process and
outcomes. We collected over 14,000 raw code-centric conversation sessions
across 10 widely used LLMs, spanning 10 languages and 8 types of execution
environments. Among these conversations, we identified more than 4,700
multi-turn samples with pairwise human preferences. Further analysis uncovers
underexplored preferences of LLMs in fine-grained domains characterized by
tasks, languages, and frameworks. To systematically examine code understanding
and generation capabilities of frontier LLMs, we curated two benchmarks based
on the collected data, namely BigCodeReward and AutoCodeArena. For
BigCodeReward, we post-processed the 4,700 conversations and evaluated the
consistency between reward models and human preferences. The evaluation shows
that most LLMs have superior performance in judging coding preferences when the
execution results are available. Inspired by these findings, we propose
AutoCodeArena, an automatic Elo rating benchmark designed to assess the coding
quality of LLMs without human involvement. We find that proprietary LLMs like
GPT-5, Claude-Sonnet-4, and Claude-Opus-4 still lead in code generation
performance among recent emerging models.
BigCodeArena is an open human evaluation platform for code generation that enables real-time execution and interaction, revealing preferences and capabilities of LLMs in coding tasks.
Chatbot Arena BigCodeArena LLM-generated code code execution human evaluation BigCodeReward AutoCodeArena Elo rating benchmark code understanding code generation