SWE-QA: Can Language Models Answer Repository-level Code Questions?
Understanding and reasoning about entire software repositories is an
essential capability for intelligent software engineering tools. While existing
benchmarks such as CoSQA and CodeQA have advanced the field, they predominantly
focus on small, self-contained code snippets. These setups fail to capture the
complexity of real-world repositories, where effective understanding and
reasoning often require navigating multiple files, understanding software
architecture, and grounding answers in long-range code dependencies. In this
paper, we present SWE-QA, a repository-level code question answering (QA)
benchmark designed to facilitate research on automated QA systems in realistic
code environments. SWE-QA involves 576 high-quality question-answer pairs
spanning diverse categories, including intention understanding, cross-file
reasoning, and multi-hop dependency analysis. To construct SWE-QA, we first
crawled 77,100 GitHub issues from 11 popular repositories. Based on an analysis
of naturally occurring developer questions extracted from these issues, we
developed a two-level taxonomy of repository-level questions and constructed a
set of seed questions for each category. For each category, we manually curated
and validated questions and collected their corresponding answers. As a
prototype application, we further develop SWE-QA-Agent, an agentic framework in
which LLM agents reason and act to find answers automatically. We evaluate six
advanced LLMs on SWE-QA under various context augmentation strategies.
Experimental results highlight the promise of LLMs, particularly our
SWE-QA-Agent framework, in addressing repository-level QA, while also revealing
open challenges and pointing to future research directions.
68ccd8ed3df9ac65e93dc6f6
Weihan Peng
68ccd8ed3df9ac65e93dc6f7
Yuling Shi
claimed_verified
2025-09-19T06:47:58.582Z
645b0c3ec35da9c7afd95421
https://cdn-avatars.huggingface.co/v1/production/uploads/645b0c3ec35da9c7afd95421/vYBrCDagHsXAo6J2p-uG0.jpeg
Yuling
user
YerbaPage
68ccd8ed3df9ac65e93dc6f8
Yuhang Wang
68ccd8ed3df9ac65e93dc6f9
Xinyun Zhang
68ccd8ed3df9ac65e93dc6fa
Beijun Shen
68ccd8ed3df9ac65e93dc6fb
Xiaodong Gu
SWE-QA: Can Language Models Answer Repository-level Code Questions?
Understanding and reasoning about entire software repositories is an
essential capability for intelligent software engineering tools. While existing
benchmarks such as CoSQA and CodeQA have advanced the field, they predominantly
focus on small, self-contained code snippets. These setups fail to capture the
complexity of real-world repositories, where effective understanding and
reasoning often require navigating multiple files, understanding software
architecture, and grounding answers in long-range code dependencies. In this
paper, we present SWE-QA, a repository-level code question answering (QA)
benchmark designed to facilitate research on automated QA systems in realistic
code environments. SWE-QA involves 576 high-quality question-answer pairs
spanning diverse categories, including intention understanding, cross-file
reasoning, and multi-hop dependency analysis. To construct SWE-QA, we first
crawled 77,100 GitHub issues from 11 popular repositories. Based on an analysis
of naturally occurring developer questions extracted from these issues, we
developed a two-level taxonomy of repository-level questions and constructed a
set of seed questions for each category. For each category, we manually curated
and validated questions and collected their corresponding answers. As a
prototype application, we further develop SWE-QA-Agent, an agentic framework in
which LLM agents reason and act to find answers automatically. We evaluate six
advanced LLMs on SWE-QA under various context augmentation strategies.
Experimental results highlight the promise of LLMs, particularly our
SWE-QA-Agent framework, in addressing repository-level QA, while also revealing
open challenges and pointing to future research directions.
SWE-QA is a repository-level code question answering benchmark that addresses the complexity of real-world software repositories, evaluated using advanced LLMs and an agentic framework.
code question answering repository-level QA software repositories cross-file reasoning multi-hop dependency analysis LLM agents SWE-QA-Agent GitHub issues