Why Personalizing Deep Learning-Based Code Completion Tools Matters
Deep learning (DL)-based code completion tools have transformed software
development by enabling advanced code generation. These tools leverage models
trained on vast amounts of code from numerous repositories, capturing general
coding patterns. However, the impact of fine-tuning these models for specific
organizations or developers to boost their performance on such subjects remains
unexplored. In this work, we fill this gap by presenting solid empirical
evidence answering this question. More specifically, we consider 136 developers
from two organizations (Apache and Spring), two model architectures (T5 and
Code Llama), and three model sizes (60M, 750M, and 7B trainable parameters). T5
models (60M, 750M) were pre-trained and fine-tuned on over 2,000 open-source
projects, excluding the subject organizations' data, and compared against
versions fine-tuned on organization- and developer-specific datasets. For the
Code Llama model (7B), we compared the performance of the already pre-trained
model publicly available online with the same model fine-tuned via
parameter-efficient fine-tuning on organization- and developer-specific
datasets. Our results show that there is a boost in prediction capabilities
provided by both an organization-specific and a developer-specific additional
fine-tuning, with the former being particularly performant. Such a finding
generalizes across (i) the two subject organizations (i.e., Apache and Spring)
and (ii) models of completely different magnitude (from 60M to 7B trainable
parameters). Finally, we show that DL models fine-tuned on an
organization-specific dataset achieve the same completion performance of
pre-trained code models used out of the box and being sim10times larger,
with consequent savings in terms of deployment and inference cost (e.g.,
smaller GPUs needed).
67ddc4f45c0bd58dff86341d
Alessandro Giagnorio
extracted_confirmed
2025-03-21T19:59:32.283Z
663486a1f64712540644cb68
https://cdn-avatars.huggingface.co/v1/production/uploads/663486a1f64712540644cb68/YZFR41ERY6UrC6rCC6Nan.jpeg
Alessandro
user
Devy1
67ddc4f45c0bd58dff86341e
Alberto Martin-Lopez
67ddc4f45c0bd58dff86341f
Gabriele Bavota
extracted_confirmed
2025-05-06T13:41:49.887Z
6638bea59e57161faac814e7
/avatars/91375b88945af50e51b7229a789a31b8.svg
Gabriele Bavota
user
gbavota
Why Personalizing Deep Learning-Based Code Completion Tools Matters
Deep learning (DL)-based code completion tools have transformed software
development by enabling advanced code generation. These tools leverage models
trained on vast amounts of code from numerous repositories, capturing general
coding patterns. However, the impact of fine-tuning these models for specific
organizations or developers to boost their performance on such subjects remains
unexplored. In this work, we fill this gap by presenting solid empirical
evidence answering this question. More specifically, we consider 136 developers
from two organizations (Apache and Spring), two model architectures (T5 and
Code Llama), and three model sizes (60M, 750M, and 7B trainable parameters). T5
models (60M, 750M) were pre-trained and fine-tuned on over 2,000 open-source
projects, excluding the subject organizations' data, and compared against
versions fine-tuned on organization- and developer-specific datasets. For the
Code Llama model (7B), we compared the performance of the already pre-trained
model publicly available online with the same model fine-tuned via
parameter-efficient fine-tuning on organization- and developer-specific
datasets. Our results show that there is a boost in prediction capabilities
provided by both an organization-specific and a developer-specific additional
fine-tuning, with the former being particularly performant. Such a finding
generalizes across (i) the two subject organizations (i.e., Apache and Spring)
and (ii) models of completely different magnitude (from 60M to 7B trainable
parameters). Finally, we show that DL models fine-tuned on an
organization-specific dataset achieve the same completion performance of
pre-trained code models used out of the box and being sim10times larger,
with consequent savings in terms of deployment and inference cost (e.g.,
smaller GPUs needed).
Organizational and developer-specific fine-tuning of deep learning models for code completion enhances prediction capabilities and reduces deployment costs, generalizing across organizations and model sizes.
T5 Code Llama pre-trained fine-tuned parameter-efficient fine-tuning prediction capabilities deployment inference cost