BuildBench: Benchmarking LLM Agents on Compiling Real-World Open-Source
  Software
Automatically compiling open-source software (OSS) projects is a vital,
labor-intensive, and complex task, which makes it a good challenge for LLM
Agents. Existing methods rely on manually curated rules and workflows, which
cannot adapt to OSS that requires customized configuration or environment
setup. Recent attempts using Large Language Models (LLMs) used selective
evaluation on a subset of highly rated OSS, a practice that underestimates the
realistic challenges of OSS compilation. In practice, compilation instructions
are often absent, dependencies are undocumented, and successful builds may even
require patching source files or modifying build scripts. We propose a more
challenging and realistic benchmark, BUILD-BENCH, comprising OSS that are more
diverse in quality, scale, and characteristics. Furthermore, we propose a
strong baseline LLM-based agent, OSS-BUILD-AGENT, an effective system with
enhanced build instruction retrieval module that achieves state-of-the-art
performance on BUILD-BENCH and is adaptable to heterogeneous OSS
characteristics. We also provide detailed analysis regarding different
compilation method design choices and their influence to the whole task,
offering insights to guide future advances. We believe performance on
BUILD-BENCH can faithfully reflect an agent's ability to tackle compilation as
a complex software engineering tasks, and, as such, our benchmark will spur
innovation with a significant impact on downstream applications in the fields
of software development and software security.
68dcfc72e795697646c0a026
Zehua Zhang
claimed_verified
2025-10-02T03:08:47.871Z
651dab8e04acc4bb90e8b8c0
/avatars/bf59ff21b4734d7bddc8751654ed2353.svg
ZEHUA ZHANG
user
STEVENZHANG904
68dcfc72e795697646c0a027
Ati Priya Bajaj
68dcfc72e795697646c0a028
Divij Handa
claimed_verified
2025-10-01T10:20:17.955Z
636f4ad38305bc19758bfb70
/avatars/7becbb5d0280114bcbf05a9604f5de1f.svg
Divij Handa
user
Divij
68dcfc72e795697646c0a029
Siyu Liu
68dcfc72e795697646c0a02a
Arvind S Raj
68dcfc72e795697646c0a02b
Hongkai Chen
68dcfc72e795697646c0a02c
Hulin Wang
68dcfc72e795697646c0a02d
Yibo Liu
68dcfc72e795697646c0a02e
Zion Leonahenahe Basque
68dcfc72e795697646c0a02f
Souradip Nath
68dcfc72e795697646c0a030
Vishal Juneja
68dcfc72e795697646c0a031
Nikhil Chapre
68dcfc72e795697646c0a032
Yan Shoshitaishvili
68dcfc72e795697646c0a033
Adam Doupé
68dcfc72e795697646c0a034
Chitta Baral
68dcfc72e795697646c0a035
Ruoyu Wang
BuildBench: Benchmarking LLM Agents on Compiling Real-World Open-Source
  Software
Automatically compiling open-source software (OSS) projects is a vital,
labor-intensive, and complex task, which makes it a good challenge for LLM
Agents. Existing methods rely on manually curated rules and workflows, which
cannot adapt to OSS that requires customized configuration or environment
setup. Recent attempts using Large Language Models (LLMs) used selective
evaluation on a subset of highly rated OSS, a practice that underestimates the
realistic challenges of OSS compilation. In practice, compilation instructions
are often absent, dependencies are undocumented, and successful builds may even
require patching source files or modifying build scripts. We propose a more
challenging and realistic benchmark, BUILD-BENCH, comprising OSS that are more
diverse in quality, scale, and characteristics. Furthermore, we propose a
strong baseline LLM-based agent, OSS-BUILD-AGENT, an effective system with
enhanced build instruction retrieval module that achieves state-of-the-art
performance on BUILD-BENCH and is adaptable to heterogeneous OSS
characteristics. We also provide detailed analysis regarding different
compilation method design choices and their influence to the whole task,
offering insights to guide future advances. We believe performance on
BUILD-BENCH can faithfully reflect an agent's ability to tackle compilation as
a complex software engineering tasks, and, as such, our benchmark will spur
innovation with a significant impact on downstream applications in the fields
of software development and software security.
A new benchmark, BUILD-BENCH, and an LLM-based agent, OSS-BUILD-AGENT, address the complexities of compiling diverse open-source software projects.
LLM Agents Large Language Models BUILD-BENCH OSS-BUILD-AGENT build instruction retrieval software engineering tasks