RepoFusion: Training Code Models to Understand Your Repository
Despite the huge success of Large Language Models (LLMs) in coding assistants
like GitHub Copilot, these models struggle to understand the context present in
the repository (e.g., imports, parent classes, files with similar names, etc.),
thereby producing inaccurate code completions. This effect is more pronounced
when using these assistants for repositories that the model has not seen during
training, such as proprietary software or work-in-progress code projects.
Recent work has shown the promise of using context from the repository during
inference. In this work, we extend this idea and propose RepoFusion, a
framework to train models to incorporate relevant repository context.
Experiments on single-line code completion show that our models trained with
repository context significantly outperform much larger code models as
CodeGen-16B-multi (sim73times larger) and closely match the performance of
the sim 70times larger StarCoderBase model that was trained with the
Fill-in-the-Middle objective. We find these results to be a novel and
compelling demonstration of the gains that training with repository context can
bring. We carry out extensive ablation studies to investigate the impact of
design choices such as context type, number of contexts, context length, and
initialization within our framework. Lastly, we release Stack-Repo, a dataset
of 200 Java repositories with permissive licenses and near-deduplicated files
that are augmented with three types of repository contexts. Additionally, we
are making available the code and trained checkpoints for our work. Our
released resources can be found at https://huggingface.co/RepoFusion.
649268cecd6ab684bd81183a
Disha Shrivastava
admin_assigned
2023-06-21T15:40:34.994Z
6438284b819f3ab20d185a3b
https://cdn-avatars.huggingface.co/v1/production/uploads/6438284b819f3ab20d185a3b/b29SoH7_Bji6GfCszjqOE.jpeg
Disha Shrivastava
user
shrivasd
649268cecd6ab684bd81183b
Denis Kocetkov
admin_assigned
2023-06-21T15:41:09.501Z
62dea51feee79ca5ef8f834a
/avatars/17e9356625785622762874fe96900599.svg
Denis Kocetkov
user
denisko
649268cecd6ab684bd81183c
Harm de Vries
admin_assigned
2023-06-21T15:41:33.325Z
62ce91e8b5811ba53c0ce7b9
https://cdn-avatars.huggingface.co/v1/production/uploads/62ce91e8b5811ba53c0ce7b9/ZNgd-p0h-q1j374QqEGrb.jpeg
Harm de Vries
user
harmdevries
649268cecd6ab684bd81183d
Dzmitry Bahdanau
649268cecd6ab684bd81183e
Torsten Scholak
admin_assigned
2023-06-21T15:43:04.731Z
60ecaa5efee13fee7ada7af4
https://cdn-avatars.huggingface.co/v1/production/uploads/1637694419284-60ecaa5efee13fee7ada7af4.jpeg
Torsten Scholak
user
tscholak
RepoFusion: Training Code Models to Understand Your Repository
Despite the huge success of Large Language Models (LLMs) in coding assistants
like GitHub Copilot, these models struggle to understand the context present in
the repository (e.g., imports, parent classes, files with similar names, etc.),
thereby producing inaccurate code completions. This effect is more pronounced
when using these assistants for repositories that the model has not seen during
training, such as proprietary software or work-in-progress code projects.
Recent work has shown the promise of using context from the repository during
inference. In this work, we extend this idea and propose RepoFusion, a
framework to train models to incorporate relevant repository context.
Experiments on single-line code completion show that our models trained with
repository context significantly outperform much larger code models as
CodeGen-16B-multi (sim73times larger) and closely match the performance of
the sim 70times larger StarCoderBase model that was trained with the
Fill-in-the-Middle objective. We find these results to be a novel and
compelling demonstration of the gains that training with repository context can
bring. We carry out extensive ablation studies to investigate the impact of
design choices such as context type, number of contexts, context length, and
initialization within our framework. Lastly, we release Stack-Repo, a dataset
of 200 Java repositories with permissive licenses and near-deduplicated files
that are augmented with three types of repository contexts. Additionally, we
are making available the code and trained checkpoints for our work. Our
released resources can be found at https://huggingface.co/RepoFusion.
RepoFusion improves code completion by incorporating repository context, leading to better performance on unseen repositories compared to larger models.
Large Language Models LLMs code assistants GitHub Copilot context repository imports parent classes files context from the repository inference framework code completion experiments CodeGen StarCoderBase Fill-in-the-Middle ablation studies context type context length initialization Stack-Repo Java repositories dataset trained checkpoints