Magicoder: Source Code Is All You Need
We introduce Magicoder, a series of fully open-source (code, weights, and
data) Large Language Models (LLMs) for code that significantly closes the gap
with top code models while having no more than 7B parameters. Magicoder models
are trained on 75K synthetic instruction data using OSS-Instruct, a novel
approach to enlightening LLMs with open-source code snippets to generate
high-quality instruction data for code. Our main motivation is to mitigate the
inherent bias of the synthetic data generated by LLMs by empowering them with a
wealth of open-source references for the production of more diverse, realistic,
and controllable data. The orthogonality of OSS-Instruct and other data
generation methods like Evol-Instruct further enables us to build an enhanced
MagicoderS. Both Magicoder and MagicoderS substantially outperform
state-of-the-art code models with similar or even larger sizes on a wide range
of coding benchmarks, including Python text-to-code generation, multilingual
coding, and data-science program completion. Notably, MagicoderS-CL-7B based on
CodeLlama even surpasses the prominent ChatGPT on HumanEval+ (66.5 vs. 65.9 in
pass@1). Overall, OSS-Instruct opens a new direction for low-bias and
high-quality instruction tuning using abundant open-source references.
656e8cd31677b468ff6b8379
Yuxiang Wei
claimed_verified
2023-12-05T08:40:19.196Z
632a176259950c1d279d5ea7
https://cdn-avatars.huggingface.co/v1/production/uploads/632a176259950c1d279d5ea7/xsSGhBXalt9RaKzSKY8uk.jpeg
Yuxiang Wei
user
yuxiang630
656e8cd31677b468ff6b837a
Zhe Wang
claimed_verified
2023-12-05T14:13:33.245Z
637bc237be63f60f4ffe2e67
https://cdn-avatars.huggingface.co/v1/production/uploads/637bc237be63f60f4ffe2e67/jXW-OqZQfedZ7dTGFlksP.png
Zhe Wang
user
zhewang
656e8cd31677b468ff6b837b
Jiawei Liu
admin_assigned
2023-12-05T08:47:19.051Z
644b584a9279988e0cbeb664
https://cdn-avatars.huggingface.co/v1/production/uploads/644b584a9279988e0cbeb664/fhWCI_Q26tTruhdFkjejw.jpeg
Jiawei Liu
user
ganler
656e8cd31677b468ff6b837c
Yifeng Ding
admin_assigned
2023-12-05T08:47:26.751Z
64ff904013f154652687bd80
/avatars/a04deda1d5c9db0c15a9311806a4b2dc.svg
Yifeng Ding
user
YifengDing
656e8cd31677b468ff6b837d
Lingming Zhang
claimed_verified
2023-12-06T10:25:42.077Z
656f473c14fa8cfccd14559e
/avatars/8f4fef3d835a7a11c2ab66dbf04f3424.svg
Lingming Zhang
user
lingming
Magicoder: Source Code Is All You Need
We introduce Magicoder, a series of fully open-source (code, weights, and
data) Large Language Models (LLMs) for code that significantly closes the gap
with top code models while having no more than 7B parameters. Magicoder models
are trained on 75K synthetic instruction data using OSS-Instruct, a novel
approach to enlightening LLMs with open-source code snippets to generate
high-quality instruction data for code. Our main motivation is to mitigate the
inherent bias of the synthetic data generated by LLMs by empowering them with a
wealth of open-source references for the production of more diverse, realistic,
and controllable data. The orthogonality of OSS-Instruct and other data
generation methods like Evol-Instruct further enables us to build an enhanced
MagicoderS. Both Magicoder and MagicoderS substantially outperform
state-of-the-art code models with similar or even larger sizes on a wide range
of coding benchmarks, including Python text-to-code generation, multilingual
coding, and data-science program completion. Notably, MagicoderS-CL-7B based on
CodeLlama even surpasses the prominent ChatGPT on HumanEval+ (66.5 vs. 65.9 in
pass@1). Overall, OSS-Instruct opens a new direction for low-bias and
high-quality instruction tuning using abundant open-source references.
Magicoder, using OSS-Instruct to incorporate open-source code snippets, achieves superior performance on coding benchmarks while reducing bias in synthetic data generation.
Large Language Models LLMs OSS-Instruct Evol-Instruct CodeLlama HumanEval+ text-to-code generation multilingual coding data-science program completion