Arctic-SnowCoder: Demystifying High-Quality Data in Code Pretraining
Recent studies have been increasingly demonstrating that high-quality data is
crucial for effective pretraining of language models. However, the precise
definition of "high-quality" remains underexplored. Focusing on the code
domain, we introduce Arctic-SnowCoder-1.3B, a data-efficient base code model
pretrained on 555B tokens through three phases of progressively refined data:
(1) general pretraining with 500B standard-quality code tokens, preprocessed
through basic filtering, deduplication, and decontamination, (2) continued
pretraining with 50B high-quality tokens, selected from phase one by a
BERT-style quality annotator trained to distinguish good code from random data,
using positive examples drawn from high-quality code files, along with
instruction data from Magicoder and StarCoder2-Instruct, and (3) enhanced
pretraining with 5B synthetic data created by Llama-3.1-70B using phase two
data as seeds, adapting the Magicoder approach for pretraining. Despite being
trained on a limited dataset, Arctic-SnowCoder achieves state-of-the-art
performance on BigCodeBench, a coding benchmark focusing on practical and
challenging programming tasks, compared to similarly sized models trained on no
more than 1T tokens, outperforming Phi-1.5-1.3B by 36%. Across all evaluated
benchmarks, Arctic-SnowCoder-1.3B beats StarCoderBase-3B pretrained on 1T
tokens. Additionally, it matches the performance of leading small base code
models trained on trillions of tokens. For example, Arctic-SnowCoder-1.3B
surpasses StarCoder2-3B, pretrained on over 3.3T tokens, on HumanEval+, a
benchmark that evaluates function-level code generation, and remains
competitive on BigCodeBench. Our evaluation presents a comprehensive analysis
justifying various design choices for Arctic-SnowCoder. Most importantly, we
find that the key to high-quality data is its alignment with the distribution
of downstream applications.
66d9403866e970a15fff17d3
Yuxiang Wei
admin_assigned
2024-09-05T08:55:03.115Z
632a176259950c1d279d5ea7
https://cdn-avatars.huggingface.co/v1/production/uploads/632a176259950c1d279d5ea7/xsSGhBXalt9RaKzSKY8uk.jpeg
Yuxiang Wei
user
yuxiang630
66d9403866e970a15fff17d4
Hojae Han
admin_assigned
2024-09-05T08:55:09.640Z
64ef02b9d2b65a3b4b7790ec
/avatars/2c65a3e65ac1980cfb46340fab51b0a3.svg
Hojae Han
user
stovecat
66d9403866e970a15fff17d5
Rajhans Samdani
admin_assigned
2024-09-05T08:55:16.759Z
64cae133bb5d195b990ad4be
/avatars/31f9f298c756b93871f4c011042f0abe.svg
Rajhans Samdani
user
rajhans-snowflake
Arctic-SnowCoder: Demystifying High-Quality Data in Code Pretraining
Recent studies have been increasingly demonstrating that high-quality data is
crucial for effective pretraining of language models. However, the precise
definition of "high-quality" remains underexplored. Focusing on the code
domain, we introduce Arctic-SnowCoder-1.3B, a data-efficient base code model
pretrained on 555B tokens through three phases of progressively refined data:
(1) general pretraining with 500B standard-quality code tokens, preprocessed
through basic filtering, deduplication, and decontamination, (2) continued
pretraining with 50B high-quality tokens, selected from phase one by a
BERT-style quality annotator trained to distinguish good code from random data,
using positive examples drawn from high-quality code files, along with
instruction data from Magicoder and StarCoder2-Instruct, and (3) enhanced
pretraining with 5B synthetic data created by Llama-3.1-70B using phase two
data as seeds, adapting the Magicoder approach for pretraining. Despite being
trained on a limited dataset, Arctic-SnowCoder achieves state-of-the-art
performance on BigCodeBench, a coding benchmark focusing on practical and
challenging programming tasks, compared to similarly sized models trained on no
more than 1T tokens, outperforming Phi-1.5-1.3B by 36%. Across all evaluated
benchmarks, Arctic-SnowCoder-1.3B beats StarCoderBase-3B pretrained on 1T
tokens. Additionally, it matches the performance of leading small base code
models trained on trillions of tokens. For example, Arctic-SnowCoder-1.3B
surpasses StarCoder2-3B, pretrained on over 3.3T tokens, on HumanEval+, a
benchmark that evaluates function-level code generation, and remains
competitive on BigCodeBench. Our evaluation presents a comprehensive analysis
justifying various design choices for Arctic-SnowCoder. Most importantly, we
find that the key to high-quality data is its alignment with the distribution
of downstream applications.
Arctic-SnowCoder-1.3B, a data-efficient code model pretrained on progressively refined high-quality data, outperforms larger models on coding benchmarks by aligning data with downstream applications.
code domain data-efficient pretraining BERT-style quality annotator synthetic data BigCodeBench StarCoderBase-3B StarCoder2-3B HumanEval+ function-level code generation