On Pretraining for Project-Level Code Completion
Repository-level pretraining is commonly used to enable large language models
for code to leverage codebase-wide context. This enhances their ability to
generate accurate and context-aware code completions. In this work, we
investigate how different repository-processing strategies affect in-context
learning in OpenCoder, a 1.5B-parameter model. We extend its context window
from 4,096 to 16,384 tokens by training on additional 1B tokens of curated
repository-level data. Despite relying on a smaller dataset than competing
models (which often use hundreds of billions of tokens), our model achieves
comparable performance on the Long Code Arena benchmark. We find that various
repository-processing techniques yield similarly strong results, with the
primary gain coming from adapting to a new rotary positional embedding (RoPE)
scaling parameter. Finally, we show that a simpler file-level training approach
at the original sequence length remains highly effective, opening up
repository-level code completion research to settings with more constrained
data and compute resources.
68f21e48e624abe1d1f0ff37
Maksim Sapronov
68f21e48e624abe1d1f0ff38
Evgeniy Glukhov
claimed_verified
2025-10-24T16:14:21.123Z
6440ff39ad24e9b2cfba8575
https://cdn-avatars.huggingface.co/v1/production/uploads/6440ff39ad24e9b2cfba8575/QoQC60vbfd2YzTtPAXpM-.jpeg
Evgeniy Glukhov
user
jenyag
On Pretraining for Project-Level Code Completion
Repository-level pretraining is commonly used to enable large language models
for code to leverage codebase-wide context. This enhances their ability to
generate accurate and context-aware code completions. In this work, we
investigate how different repository-processing strategies affect in-context
learning in OpenCoder, a 1.5B-parameter model. We extend its context window
from 4,096 to 16,384 tokens by training on additional 1B tokens of curated
repository-level data. Despite relying on a smaller dataset than competing
models (which often use hundreds of billions of tokens), our model achieves
comparable performance on the Long Code Arena benchmark. We find that various
repository-processing techniques yield similarly strong results, with the
primary gain coming from adapting to a new rotary positional embedding (RoPE)
scaling parameter. Finally, we show that a simpler file-level training approach
at the original sequence length remains highly effective, opening up
repository-level code completion research to settings with more constrained
data and compute resources.
Extending the context window and adapting to a new rotary positional embedding scaling parameter improve repository-level code completion in OpenCoder, achieving performance comparable to larger models with less data.
repository-level pretraining code completions OpenCoder context window rotary positional embedding (RoPE) Long Code Arena benchmark file-level training