To Code, or Not To Code? Exploring Impact of Code in Pre-training
Including code in the pre-training data mixture, even for models not
specifically designed for code, has become a common practice in LLMs
pre-training. While there has been anecdotal consensus among practitioners that
code data plays a vital role in general LLMs' performance, there is only
limited work analyzing the precise impact of code on non-code tasks. In this
work, we systematically investigate the impact of code data on general
performance. We ask "what is the impact of code data used in pre-training on a
large variety of downstream tasks beyond code generation". We conduct extensive
ablations and evaluate across a broad range of natural language reasoning
tasks, world knowledge tasks, code benchmarks, and LLM-as-a-judge win-rates for
models with sizes ranging from 470M to 2.8B parameters. Across settings, we
find a consistent results that code is a critical building block for
generalization far beyond coding tasks and improvements to code quality have an
outsized impact across all tasks. In particular, compared to text-only
pre-training, the addition of code results in up to relative increase of 8.2%
in natural language (NL) reasoning, 4.2% in world knowledge, 6.6% improvement
in generative win-rates, and a 12x boost in code performance respectively. Our
work suggests investments in code quality and preserving code during
pre-training have positive impacts.
66c556dd50f505c5451ce8e1
Viraat Aryabumi
claimed_verified
2024-08-22T19:51:19.578Z
633e9b78acd351fc37d69560
https://cdn-avatars.huggingface.co/v1/production/uploads/633e9b78acd351fc37d69560/rCk8z-C4oOlrksp-QE2lV.jpeg
Viraat Aryabumi
user
viraat
66c556dd50f505c5451ce8e2
Yixuan Su
66c556dd50f505c5451ce8e3
Raymond Ma
66c556dd50f505c5451ce8e4
Adrien Morisot
66c556dd50f505c5451ce8e5
Ivan Zhang
66c556dd50f505c5451ce8e6
Acyr Locatelli
66c556dd50f505c5451ce8e7
Marzieh Fadaee
claimed_verified
2024-11-15T14:10:34.795Z
6441042d5d600fb0951a5f99
https://cdn-avatars.huggingface.co/v1/production/uploads/6441042d5d600fb0951a5f99/4CbOaYcEz99BtVAQvnGTn.jpeg
Marzieh Fadaee
user
MarziehFadaee
66c556dd50f505c5451ce8e8
Ahmet Üstün
66c556dd50f505c5451ce8e9
Sara Hooker
claimed_verified
2024-09-03T07:48:46.677Z
63434eb76f59b79da07dbddf
https://cdn-avatars.huggingface.co/v1/production/uploads/63434eb76f59b79da07dbddf/BEwmVjqPNYlqmutXG0G6e.jpeg
Sara Hooker
user
sarahooker
To Code, or Not To Code? Exploring Impact of Code in Pre-training
Including code in the pre-training data mixture, even for models not
specifically designed for code, has become a common practice in LLMs
pre-training. While there has been anecdotal consensus among practitioners that
code data plays a vital role in general LLMs' performance, there is only
limited work analyzing the precise impact of code on non-code tasks. In this
work, we systematically investigate the impact of code data on general
performance. We ask "what is the impact of code data used in pre-training on a
large variety of downstream tasks beyond code generation". We conduct extensive
ablations and evaluate across a broad range of natural language reasoning
tasks, world knowledge tasks, code benchmarks, and LLM-as-a-judge win-rates for
models with sizes ranging from 470M to 2.8B parameters. Across settings, we
find a consistent results that code is a critical building block for
generalization far beyond coding tasks and improvements to code quality have an
outsized impact across all tasks. In particular, compared to text-only
pre-training, the addition of code results in up to relative increase of 8.2%
in natural language (NL) reasoning, 4.2% in world knowledge, 6.6% improvement
in generative win-rates, and a 12x boost in code performance respectively. Our
work suggests investments in code quality and preserving code during
pre-training have positive impacts.
The inclusion of code data during pre-training significantly improves LLMs' performance across various natural language and world knowledge tasks, beyond code generation.
LLMs (Large Language Models) code data pre-training natural language reasoning world knowledge tasks code benchmarks LLM-as-a-judge generative win-rates