OctoPack: Instruction Tuning Code Large Language Models
Finetuning large language models (LLMs) on instructions leads to vast
performance improvements on natural language tasks. We apply instruction tuning
using code, leveraging the natural structure of Git commits, which pair code
changes with human instructions. We compile CommitPack: 4 terabytes of Git
commits across 350 programming languages. We benchmark CommitPack against other
natural and synthetic code instructions (xP3x, Self-Instruct, OASST) on the 16B
parameter StarCoder model, and achieve state-of-the-art performance among
models not trained on OpenAI outputs, on the HumanEval Python benchmark (46.2%
pass@1). We further introduce HumanEvalPack, expanding the HumanEval benchmark
to a total of 3 coding tasks (Code Repair, Code Explanation, Code Synthesis)
across 6 languages (Python, JavaScript, Java, Go, C++, Rust). Our models,
OctoCoder and OctoGeeX, achieve the best performance across HumanEvalPack among
all permissive models, demonstrating CommitPack's benefits in generalizing to a
wider set of languages and natural coding tasks. Code, models and data are
freely available at https://github.com/bigcode-project/octopack.
64dacfd68da011d65603631c
Niklas Muennighoff
extracted_confirmed
2023-08-15T01:19:14.006Z
5f1eb362eec0ad2a071ad6e2
https://cdn-avatars.huggingface.co/v1/production/uploads/5f1eb362eec0ad2a071ad6e2/IXMYkYKuTwn6kBdWnQeeY.png
Niklas Muennighoff
user
Muennighoff
64dacfd68da011d65603631d
Qian Liu
claimed_verified
2023-08-17T13:53:46.154Z
612ee6a7b960e78c6d2319d4
https://cdn-avatars.huggingface.co/v1/production/uploads/612ee6a7b960e78c6d2319d4/2Hu9BaAyXbyh1vt0v1Qui.jpeg
Qian Liu
user
SivilTaram
64dacfd68da011d65603631e
Armel Zebaze
64dacfd68da011d65603631f
Qinkai Zheng
admin_assigned
2023-08-15T12:57:37.981Z
6231576e92e83fd1179ac3f0
https://cdn-avatars.huggingface.co/v1/production/uploads/1664543160657-6231576e92e83fd1179ac3f0.jpeg
Qinkai Zheng
user
Stanislas
64dacfd68da011d656036320
Binyuan Hui
admin_assigned
2023-08-15T12:57:56.438Z
61e4c4ca1ab24785ac11ba69
https://cdn-avatars.huggingface.co/v1/production/uploads/61e4c4ca1ab24785ac11ba69/1Q1zhhyGSJ9RJG9MzwxVv.jpeg
Binyuan Hui
user
huybery
64dacfd68da011d656036321
Terry Yue Zhuo
admin_assigned
2023-08-15T12:58:05.766Z
62b7fb545233925f253531c8
https://cdn-avatars.huggingface.co/v1/production/uploads/62b7fb545233925f253531c8/W50u2G1HK3EtUKHRU189V.jpeg
Terry Yue Zhuo
user
terryyz
64dacfd68da011d656036322
Swayam Singh
admin_assigned
2023-08-15T12:58:25.286Z
62f47c093561a52aa5a67c90
https://cdn-avatars.huggingface.co/v1/production/uploads/62f47c093561a52aa5a67c90/d4sFnllrLH5BWbZDRNvMn.jpeg
Swayam Singh
user
rootacess
64dacfd68da011d656036323
Xiangru Tang
admin_assigned
2023-08-15T12:58:40.805Z
64709e80d9360cd9d8e7073d
/avatars/ec37977b9a10e823e0d5bb73d152f253.svg
tang xiangrun
user
stxrun
64dacfd68da011d656036324
Leandro von Werra
admin_assigned
2023-08-15T12:59:12.417Z
5e48005437cb5b49818287a5
https://cdn-avatars.huggingface.co/v1/production/uploads/5e48005437cb5b49818287a5/4uCXGGui-9QifAT4qelxU.png
Leandro von Werra
user
lvwerra
64dacfd68da011d656036325
Shayne Longpre
admin_assigned
2023-08-15T12:59:22.573Z
61f4283a81c4d30f58140242
/avatars/a1cf1ef1fd442c36ed65c68e51919fed.svg
Shayne Longpre
user
Shayne
OctoPack: Instruction Tuning Code Large Language Models
Finetuning large language models (LLMs) on instructions leads to vast
performance improvements on natural language tasks. We apply instruction tuning
using code, leveraging the natural structure of Git commits, which pair code
changes with human instructions. We compile CommitPack: 4 terabytes of Git
commits across 350 programming languages. We benchmark CommitPack against other
natural and synthetic code instructions (xP3x, Self-Instruct, OASST) on the 16B
parameter StarCoder model, and achieve state-of-the-art performance among
models not trained on OpenAI outputs, on the HumanEval Python benchmark (46.2%
pass@1). We further introduce HumanEvalPack, expanding the HumanEval benchmark
to a total of 3 coding tasks (Code Repair, Code Explanation, Code Synthesis)
across 6 languages (Python, JavaScript, Java, Go, C++, Rust). Our models,
OctoCoder and OctoGeeX, achieve the best performance across HumanEvalPack among
all permissive models, demonstrating CommitPack's benefits in generalizing to a
wider set of languages and natural coding tasks. Code, models and data are
freely available at https://github.com/bigcode-project/octopack.
Instruction tuning using Git commits improves performance on natural language and coding tasks compared to other benchmarks, with models achieving state-of-the-art results on expanded HumanEvalPack.
instruction tuning code Git commits CommitPack 16B parameter StarCoder HumanEval benchmark HumanEvalPack Code Repair Code Explanation Code Synthesis OctoCoder OctoGeeX