LoCoBench: A Benchmark for Long-Context Large Language Models in Complex
  Software Engineering
The emergence of long-context language models with context windows extending
to millions of tokens has created new opportunities for sophisticated code
understanding and software development evaluation. We propose LoCoBench, a
comprehensive benchmark specifically designed to evaluate long-context LLMs in
realistic, complex software development scenarios. Unlike existing code
evaluation benchmarks that focus on single-function completion or short-context
tasks, LoCoBench addresses the critical evaluation gap for long-context
capabilities that require understanding entire codebases, reasoning across
multiple files, and maintaining architectural consistency across large-scale
software systems. Our benchmark provides 8,000 evaluation scenarios
systematically generated across 10 programming languages, with context lengths
spanning 10K to 1M tokens, a 100x variation that enables precise assessment of
long-context performance degradation in realistic software development
settings. LoCoBench introduces 8 task categories that capture essential
long-context capabilities: architectural understanding, cross-file refactoring,
multi-session development, bug investigation, feature implementation, code
comprehension, integration testing, and security analysis. Through a 5-phase
pipeline, we create diverse, high-quality scenarios that challenge LLMs to
reason about complex codebases at unprecedented scale. We introduce a
comprehensive evaluation framework with 17 metrics across 4 dimensions,
including 8 new evaluation metrics, combined in a LoCoBench Score (LCBS). Our
evaluation of state-of-the-art long-context models reveals substantial
performance gaps, demonstrating that long-context understanding in complex
software development represents a significant unsolved challenge that demands
more attention. LoCoBench is released at:
https://github.com/SalesforceAIResearch/LoCoBench.
68c381befc1747b9124039f9
Jielin Qiu
claimed_verified
2025-09-18T13:27:55.602Z
63294eeace50f125375243c1
/avatars/1a5e3b5552e9127d4e14d85880b122bd.svg
Jielin Qiu
user
jasonqiu
68c381befc1747b9124039fa
Zuxin Liu
68c381befc1747b9124039fb
Zhiwei Liu
68c381befc1747b9124039fc
Rithesh Murthy
68c381befc1747b9124039fd
Jianguo Zhang
68c381befc1747b9124039fe
Haolin Chen
claimed_verified
2025-10-10T10:00:27.988Z
65e0c21ebd0c59bbf271004c
/avatars/3210f41fd433d798fd7857c943288625.svg
Haolin Chen
user
hlnchen
68c381befc1747b9124039ff
Shiyu Wang
68c381befc1747b912403a00
Ming Zhu
68c381befc1747b912403a01
Liangwei Yang
68c381befc1747b912403a02
Juntao Tan
68c381befc1747b912403a03
Zhepeng Cen
claimed_verified
2025-10-08T08:04:07.209Z
64bb3e754b4ff0d509141938
/avatars/81617fd676a3d3133811348055e68fc7.svg
Zhepeng Cen
user
czp16
68c381befc1747b912403a04
Cheng Qian
claimed_verified
2025-09-29T06:04:45.305Z
665e121c6007027038fd4005
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/sIVBJAGM-Kneq9KMf8aXb.png
Cheng Qian
user
chengq9
68c381befc1747b912403a05
Shelby Heinecke
68c381befc1747b912403a06
Weiran Yao
claimed_verified
2025-10-08T08:04:09.327Z
661573234c2f29635e93bb71
/avatars/fba95e382454485766b6349d6281b715.svg
Weiran Yao
user
weiranyao
68c381befc1747b912403a07
Silvio Savarese
68c381befc1747b912403a08
Caiming Xiong
68c381befc1747b912403a09
Huan Wang
LoCoBench: A Benchmark for Long-Context Large Language Models in Complex
  Software Engineering
The emergence of long-context language models with context windows extending
to millions of tokens has created new opportunities for sophisticated code
understanding and software development evaluation. We propose LoCoBench, a
comprehensive benchmark specifically designed to evaluate long-context LLMs in
realistic, complex software development scenarios. Unlike existing code
evaluation benchmarks that focus on single-function completion or short-context
tasks, LoCoBench addresses the critical evaluation gap for long-context
capabilities that require understanding entire codebases, reasoning across
multiple files, and maintaining architectural consistency across large-scale
software systems. Our benchmark provides 8,000 evaluation scenarios
systematically generated across 10 programming languages, with context lengths
spanning 10K to 1M tokens, a 100x variation that enables precise assessment of
long-context performance degradation in realistic software development
settings. LoCoBench introduces 8 task categories that capture essential
long-context capabilities: architectural understanding, cross-file refactoring,
multi-session development, bug investigation, feature implementation, code
comprehension, integration testing, and security analysis. Through a 5-phase
pipeline, we create diverse, high-quality scenarios that challenge LLMs to
reason about complex codebases at unprecedented scale. We introduce a
comprehensive evaluation framework with 17 metrics across 4 dimensions,
including 8 new evaluation metrics, combined in a LoCoBench Score (LCBS). Our
evaluation of state-of-the-art long-context models reveals substantial
performance gaps, demonstrating that long-context understanding in complex
software development represents a significant unsolved challenge that demands
more attention. LoCoBench is released at:
https://github.com/SalesforceAIResearch/LoCoBench.
LoCoBench evaluates long-context language models in complex software development scenarios, addressing the gap in understanding entire codebases and maintaining architectural consistency across large-scale systems.
long-context language models LoCoBench code evaluation benchmarks long-context capabilities codebases cross-file refactoring multi-session development bug investigation feature implementation code comprehension integration testing security analysis LoCoBench Score (LCBS)