Diff-XYZ: A Benchmark for Evaluating Diff Understanding
Reliable handling of code diffs is central to agents that edit and refactor
repositories at scale. We introduce Diff-XYZ, a compact benchmark for code-diff
understanding with three supervised tasks: apply (old code + diff
rightarrow new code), anti-apply (new code - diff rightarrow old code),
and diff generation (new code - old code rightarrow diff). Instances in
the benchmark are triples langle old code, new code,
diff rangle drawn from real commits in CommitPackFT, paired with
automatic metrics and a clear evaluation protocol. We use the benchmark to do a
focused empirical study of the unified diff format and run a cross-format
comparison of different diff representations. Our findings reveal that
different formats should be used depending on the use case and model size. For
example, representing diffs in search-replace format is good for larger models
in the diff generation scenario, yet not suited well for diff analysis and
smaller models. The Diff-XYZ benchmark is a reusable foundation for assessing
and improving diff handling in LLMs that can aid future development of diff
formats and models editing code. The dataset is published on HuggingFace Hub:
https://huggingface.co/datasets/JetBrains-Research/diff-xyz.
68f22352e624abe1d1f0ff54
Evgeniy Glukhov
claimed_verified
2025-10-24T16:14:18.740Z
6440ff39ad24e9b2cfba8575
https://cdn-avatars.huggingface.co/v1/production/uploads/6440ff39ad24e9b2cfba8575/QoQC60vbfd2YzTtPAXpM-.jpeg
Evgeniy Glukhov
user
jenyag
68f22352e624abe1d1f0ff55
Michele Conti
68f22352e624abe1d1f0ff56
Egor Bogomolov
claimed_verified
2025-10-24T16:14:14.811Z
64380bed961bb61e463bf93d
https://cdn-avatars.huggingface.co/v1/production/uploads/64380bed961bb61e463bf93d/zsel0Dzv1yU9O8zAxvCBw.jpeg
Egor Bogomolov
user
egor-bogomolov
68f22352e624abe1d1f0ff57
Yaroslav Golubev
68f22352e624abe1d1f0ff58
Alexander Bezzubov
Diff-XYZ: A Benchmark for Evaluating Diff Understanding
Reliable handling of code diffs is central to agents that edit and refactor
repositories at scale. We introduce Diff-XYZ, a compact benchmark for code-diff
understanding with three supervised tasks: apply (old code + diff
rightarrow new code), anti-apply (new code - diff rightarrow old code),
and diff generation (new code - old code rightarrow diff). Instances in
the benchmark are triples langle old code, new code,
diff rangle drawn from real commits in CommitPackFT, paired with
automatic metrics and a clear evaluation protocol. We use the benchmark to do a
focused empirical study of the unified diff format and run a cross-format
comparison of different diff representations. Our findings reveal that
different formats should be used depending on the use case and model size. For
example, representing diffs in search-replace format is good for larger models
in the diff generation scenario, yet not suited well for diff analysis and
smaller models. The Diff-XYZ benchmark is a reusable foundation for assessing
and improving diff handling in LLMs that can aid future development of diff
formats and models editing code. The dataset is published on HuggingFace Hub:
https://huggingface.co/datasets/JetBrains-Research/diff-xyz.
A benchmark for code-diff understanding with tasks including apply, anti-apply, and diff generation, revealing optimal diff formats based on model size and use case.
code-diff understanding unified diff format search-replace format diff generation diff analysis LLMs