GitChameleon: Evaluating AI Code Generation Against Python Library
  Version Incompatibilities
The rapid evolution of software libraries poses a considerable hurdle for
code generation, necessitating continuous adaptation to frequent version
updates while preserving backward compatibility. While existing code evolution
benchmarks provide valuable insights, they typically lack execution-based
evaluation for generating code compliant with specific library versions. To
address this, we introduce GitChameleon, a novel, meticulously curated dataset
comprising 328 Python code completion problems, each conditioned on specific
library versions and accompanied by executable unit tests. GitChameleon
rigorously evaluates the capacity of contemporary large language models (LLMs),
LLM-powered agents, code assistants, and RAG systems to perform
version-conditioned code generation that demonstrates functional accuracy
through execution. Our extensive evaluations indicate that state-of-the-art
systems encounter significant challenges with this task; enterprise models
achieving baseline success rates in the 48-51\% range, underscoring the
intricacy of the problem. By offering an execution-based benchmark emphasizing
the dynamic nature of code libraries, GitChameleon enables a clearer
understanding of this challenge and helps guide the development of more
adaptable and dependable AI code generation methods. We make the dataset and
evaluation code publicly available at
https://github.com/mrcabbage972/GitChameleonBenchmark.
6878bb18001546c83aa4fa92
Diganta Misra
6878bb18001546c83aa4fa93
Nizar Islah
6878bb18001546c83aa4fa94
Victor May
6878bb18001546c83aa4fa95
Brice Rauby
6878bb18001546c83aa4fa96
Zihan Wang
6878bb18001546c83aa4fa97
Justine Gehring
6878bb18001546c83aa4fa98
Antonio Orvieto
6878bb18001546c83aa4fa99
Muawiz Chaudhary
6878bb18001546c83aa4fa9a
Eilif B. Muller
6878bb18001546c83aa4fa9b
Irina Rish
6878bb18001546c83aa4fa9c
Samira Ebrahimi Kahou
6878bb18001546c83aa4fa9d
Massimo Caccia
GitChameleon: Evaluating AI Code Generation Against Python Library
  Version Incompatibilities
The rapid evolution of software libraries poses a considerable hurdle for
code generation, necessitating continuous adaptation to frequent version
updates while preserving backward compatibility. While existing code evolution
benchmarks provide valuable insights, they typically lack execution-based
evaluation for generating code compliant with specific library versions. To
address this, we introduce GitChameleon, a novel, meticulously curated dataset
comprising 328 Python code completion problems, each conditioned on specific
library versions and accompanied by executable unit tests. GitChameleon
rigorously evaluates the capacity of contemporary large language models (LLMs),
LLM-powered agents, code assistants, and RAG systems to perform
version-conditioned code generation that demonstrates functional accuracy
through execution. Our extensive evaluations indicate that state-of-the-art
systems encounter significant challenges with this task; enterprise models
achieving baseline success rates in the 48-51\% range, underscoring the
intricacy of the problem. By offering an execution-based benchmark emphasizing
the dynamic nature of code libraries, GitChameleon enables a clearer
understanding of this challenge and helps guide the development of more
adaptable and dependable AI code generation methods. We make the dataset and
evaluation code publicly available at
https://github.com/mrcabbage972/GitChameleonBenchmark.
GitChameleon is a dataset for evaluating version-conditioned code generation by large language models, LLM-powered agents, code assistants, and RAG systems using execution-based tests.
GitChameleon large language models LLM-powered agents code assistants RAG systems code completion library versions executable unit tests version-conditioned code generation functional accuracy