LoCoBench: A Benchmark for Long-Context Large Language Models in Complex
  Software Engineering
The emergence of long-context language models with context windows extending
to millions of tokens has created new opportunities for sophisticated code
understanding and software development evaluation. We propose LoCoBench, a
comprehensive benchmark specifically designed to evaluate long-context LLMs in
realistic, complex software development scenarios. Unlike existing code
evaluation benchmarks that focus on single-function completion or short-context
tasks, LoCoBench addresses the critical evaluation gap for long-context
capabilities that require understanding entire codebases, reasoning across
multiple files, and maintaining architectural consistency across large-scale
software systems. Our benchmark provides 8,000 evaluation scenarios
systematically generated across 10 programming languages, with context lengths
spanning 10K to 1M tokens, a 100x variation that enables precise assessment of
long-context performance degradation in realistic software development
settings. LoCoBench introduces 8 task categories that capture essential
long-context capabilities: architectural understanding, cross-file refactoring,
multi-session development, bug investigation, feature implementation, code
comprehension, integration testing, and security analysis. Through a 5-phase
pipeline, we create diverse, high-quality scenarios that challenge LLMs to
reason about complex codebases at unprecedented scale. We introduce a
comprehensive evaluation framework with 17 metrics across 4 dimensions,
including 8 new evaluation metrics, combined in a LoCoBench Score (LCBS). Our
evaluation of state-of-the-art long-context models reveals substantial
performance gaps, demonstrating that long-context understanding in complex
software development represents a significant unsolved challenge that demands
more attention. LoCoBench is released at:
https://github.com/SalesforceAIResearch/LoCoBench.
68c381befc1747b9124039f9
Jielin Qiu
claimed_verified
2025-09-18T13:27:55.602Z
63294eeace50f125375243c1
/avatars/1a5e3b5552e9127d4e14d85880b122bd.svg
Jielin Qiu
user
jasonqiu
68c381befc1747b9124039fa
Zuxin Liu
68c381befc1747b9124039fb
Zhiwei Liu
68c381befc1747b9124039fc
Rithesh Murthy
68c381befc1747b9124039fd
Jianguo Zhang
68c381befc1747b9124039fe
Haolin Chen
68c381befc1747b9124039ff
Shiyu Wang
68c381befc1747b912403a00
Ming Zhu
68c381befc1747b912403a01
Liangwei Yang
68c381befc1747b912403a02
Juntao Tan
68c381befc1747b912403a03
Zhepeng Cen
68c381befc1747b912403a04
Cheng Qian
68c381befc1747b912403a05
Shelby Heinecke
68c381befc1747b912403a06
Weiran Yao
68c381befc1747b912403a07
Silvio Savarese
68c381befc1747b912403a08
Caiming Xiong
68c381befc1747b912403a09
Huan Wang
LoCoBench: A Benchmark for Long-Context Large Language Models in Complex
  Software Engineering
The emergence of long-context language models with context windows extending
to millions of tokens has created new opportunities for sophisticated code
understanding and software development evaluation. We propose LoCoBench, a
comprehensive benchmark specifically designed to evaluate long-context LLMs in
realistic, complex software development scenarios. Unlike existing code
evaluation benchmarks that focus on single-function completion or short-context
tasks, LoCoBench addresses the critical evaluation gap for long-context
capabilities that require understanding entire codebases, reasoning across
multiple files, and maintaining architectural consistency across large-scale
software systems. Our benchmark provides 8,000 evaluation scenarios
systematically generated across 10 programming languages, with context lengths
spanning 10K to 1M tokens, a 100x variation that enables precise assessment of
long-context performance degradation in realistic software development
settings. LoCoBench introduces 8 task categories that capture essential
long-context capabilities: architectural understanding, cross-file refactoring,
multi-session development, bug investigation, feature implementation, code
comprehension, integration testing, and security analysis. Through a 5-phase
pipeline, we create diverse, high-quality scenarios that challenge LLMs to
reason about complex codebases at unprecedented scale. We introduce a
comprehensive evaluation framework with 17 metrics across 4 dimensions,
including 8 new evaluation metrics, combined in a LoCoBench Score (LCBS). Our
evaluation of state-of-the-art long-context models reveals substantial
performance gaps, demonstrating that long-context understanding in complex
software development represents a significant unsolved challenge that demands
more attention. LoCoBench is released at:
https://github.com/SalesforceAIResearch/LoCoBench.
LoCoBench evaluates long-context language models in complex software development scenarios, addressing the gap in understanding entire codebases and maintaining architectural consistency across large-scale systems.
long-context language models LoCoBench code evaluation benchmarks long-context capabilities codebases cross-file refactoring multi-session development bug investigation feature implementation code comprehension integration testing security analysis LoCoBench Score (LCBS)