OmniGIRL: A Multilingual and Multimodal Benchmark for GitHub Issue
  Resolution
The GitHub issue resolution task aims to resolve issues reported in
repositories automatically. With advances in large language models (LLMs), this
task has gained increasing attention, and several benchmarks are proposed to
evaluate the issue resolution ability of LLMs. However, existing benchmarks
have three main limitations. First, current benchmarks focus on a single
programming language, limiting the evaluation of issues from repositories
across different languages. Second, they usually cover a narrow range of
domains, which may fail to represent the diversity of real-world issues. Third,
existing benchmarks rely solely on textual information in issue descriptions,
overlooking multimodal information such as images in issues. In this paper, we
propose OmniGIRL, a GitHub Issue ResoLution benchmark that is multilingual,
multimodal, and multi-domain. OmniGIRL includes 959 task instances, which are
collected from repositories across four programming languages (i.e., Python,
JavaScript, TypeScript, and Java) and eight different domains. Our evaluation
shows that current LLMs show limited performances on OmniGIRL. Notably, the
best-performing model, GPT-4o, resolves only 8.6% of the issues. Besides, we
find that current LLMs struggle to resolve issues requiring understanding
images. The best performance is achieved by Claude-3.5-Sonnet, which resolves
only 10.5% of the issues with image information. Finally, we analyze the
reasons behind current LLMs' failure on OmniGIRL, providing insights for future
improvements.
681c7a87215f1831099fdfd2
Lianghong Guo
681c7a87215f1831099fdfd3
Wei Tao
claimed_verified
2025-05-08T09:58:00.562Z
6355473d525beaee688b7ba1
/avatars/1fb0d57ed5f1a9b872a1ada8b2973ffb.svg
Wei Tao
user
itaowe
681c7a87215f1831099fdfd4
Runhan Jiang
681c7a87215f1831099fdfd5
Yanlin Wang
681c7a87215f1831099fdfd6
Jiachi Chen
admin_assigned
2025-05-14T13:00:45.102Z
66ee1c302e68abec6890ac35
/avatars/c1130a51d58591edec23f10bf0291187.svg
Chen
user
Jiachi317
681c7a87215f1831099fdfd7
Xilin Liu
admin_assigned
2025-05-14T13:00:55.524Z
65efafff61b46846709541c0
/avatars/a81e7c5d517cd089b9fbd7c10fc23250.svg
Liu Xilin
user
armstrong111
681c7a87215f1831099fdfd8
Yuchi Ma
681c7a87215f1831099fdfd9
Mingzhi Mao
681c7a87215f1831099fdfda
Hongyu Zhang
681c7a87215f1831099fdfdb
Zibin Zheng
OmniGIRL: A Multilingual and Multimodal Benchmark for GitHub Issue
  Resolution
The GitHub issue resolution task aims to resolve issues reported in
repositories automatically. With advances in large language models (LLMs), this
task has gained increasing attention, and several benchmarks are proposed to
evaluate the issue resolution ability of LLMs. However, existing benchmarks
have three main limitations. First, current benchmarks focus on a single
programming language, limiting the evaluation of issues from repositories
across different languages. Second, they usually cover a narrow range of
domains, which may fail to represent the diversity of real-world issues. Third,
existing benchmarks rely solely on textual information in issue descriptions,
overlooking multimodal information such as images in issues. In this paper, we
propose OmniGIRL, a GitHub Issue ResoLution benchmark that is multilingual,
multimodal, and multi-domain. OmniGIRL includes 959 task instances, which are
collected from repositories across four programming languages (i.e., Python,
JavaScript, TypeScript, and Java) and eight different domains. Our evaluation
shows that current LLMs show limited performances on OmniGIRL. Notably, the
best-performing model, GPT-4o, resolves only 8.6% of the issues. Besides, we
find that current LLMs struggle to resolve issues requiring understanding
images. The best performance is achieved by Claude-3.5-Sonnet, which resolves
only 10.5% of the issues with image information. Finally, we analyze the
reasons behind current LLMs' failure on OmniGIRL, providing insights for future
improvements.
OmniGIRL, a multilingual, multimodal, and multi-domain GitHub issue resolution benchmark, demonstrates that current LLMs perform poorly, particularly in issues involving images.
multilingual multimodal multi-domain GitHub issue resolution benchmark programming languages domains large language models