Code Representation Learning At Scale
Recent studies have shown that code language models at scale demonstrate
significant performance gains on downstream tasks, i.e., code generation.
However, most of the existing works on code representation learning train
models at a hundred million parameter scale using very limited pretraining
corpora. In this work, we fuel code representation learning with a vast amount
of code data via a two-stage pretraining scheme. We first train the encoders
via a mix that leverages both randomness in masking language modeling and the
structure aspect of programming language. We then enhance the representations
via contrastive learning with hard negative and hard positive constructed in an
unsupervised manner. We establish an off-the-shelf encoder model that
persistently outperforms the existing models on a wide variety of downstream
tasks by large margins. To comprehend the factors contributing to successful
code representation learning, we conduct detailed ablations and share our
findings on (i) a customized and effective token-level denoising scheme for
source code; (ii) the importance of hard negatives and hard positives; (iii)
how the proposed bimodal contrastive learning boost the cross-lingual semantic
search performance; and (iv) how the pretraining schemes decide the downstream
task performance scales with the model size.
65c1b72e35cfc292c0cad02b
Dejiao Zhang
admin_assigned
2024-05-02T09:37:04.263Z
61be0f906196e18d07adbc84
/avatars/257cc25509e97c6a62e322e876a66c04.svg
Dejiao Zhang
user
dzxl1946
65c1b72e35cfc292c0cad02c
Wasi Ahmad
claimed_verified
2025-04-04T07:12:27.747Z
67ec45ba92673a326c067b52
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/PcD60kFRczOblxgc1bWcl.png
Wasi Uddin Ahmad
user
wasiuddina
65c1b72e35cfc292c0cad02d
Ming Tan
65c1b72e35cfc292c0cad02e
Hantian Ding
65c1b72e35cfc292c0cad02f
Ramesh Nallapati
65c1b72e35cfc292c0cad030
Dan Roth
65c1b72e35cfc292c0cad031
Xiaofei Ma
65c1b72e35cfc292c0cad032
Bing Xiang
Code Representation Learning At Scale
Recent studies have shown that code language models at scale demonstrate
significant performance gains on downstream tasks, i.e., code generation.
However, most of the existing works on code representation learning train
models at a hundred million parameter scale using very limited pretraining
corpora. In this work, we fuel code representation learning with a vast amount
of code data via a two-stage pretraining scheme. We first train the encoders
via a mix that leverages both randomness in masking language modeling and the
structure aspect of programming language. We then enhance the representations
via contrastive learning with hard negative and hard positive constructed in an
unsupervised manner. We establish an off-the-shelf encoder model that
persistently outperforms the existing models on a wide variety of downstream
tasks by large margins. To comprehend the factors contributing to successful
code representation learning, we conduct detailed ablations and share our
findings on (i) a customized and effective token-level denoising scheme for
source code; (ii) the importance of hard negatives and hard positives; (iii)
how the proposed bimodal contrastive learning boost the cross-lingual semantic
search performance; and (iv) how the pretraining schemes decide the downstream
task performance scales with the model size.
A two-stage pretraining scheme using a combination of masking language modeling and contrastive learning significantly improves code representation learning and performance on various downstream tasks.
masking language modeling contrastive learning hard negative hard positive bimodal contrastive learning cross-lingual semantic search token-level denoising