OpenCoder: The Open Cookbook for Top-Tier Code Large Language Models
Large language models (LLMs) for code have become indispensable in various
domains, including code generation, reasoning tasks and agent systems.While
open-access code LLMs are increasingly approaching the performance levels of
proprietary models, high-quality code LLMs suitable for rigorous scientific
investigation, particularly those with reproducible data processing pipelines
and transparent training protocols, remain limited. The scarcity is due to
various challenges, including resource constraints, ethical considerations, and
the competitive advantages of keeping models advanced. To address the gap, we
introduce OpenCoder, a top-tier code LLM that not only achieves performance
comparable to leading models but also serves as an ``open cookbook'' for the
research community. Unlike most prior efforts, we release not only model
weights and inference code, but also the reproducible training data, complete
data processing pipeline, rigorous experimental ablation results, and detailed
training protocols for open scientific research. Through this comprehensive
release, we identify the key ingredients for building a top-tier code LLM: (1)
code optimized heuristic rules for data cleaning and methods for data
deduplication, (2) recall of text corpus related to code and (3) high-quality
synthetic data in both annealing and supervised fine-tuning stages. By offering
this level of openness, we aim to broaden access to all aspects of a top-tier
code LLM, with OpenCoder serving as both a powerful model and an open
foundation to accelerate research, and enable reproducible advancements in code
AI.
672d8d720028ad1c50f6c7d2
Siming Huang
claimed_verified
2024-11-15T09:30:08.966Z
65bd066c696d46ec68aca0ae
https://cdn-avatars.huggingface.co/v1/production/uploads/65bd066c696d46ec68aca0ae/EvKLmmxMPle9hMahyj7cn.jpeg
Siming Huang
user
Simingh
672d8d720028ad1c50f6c7d3
Tianhao Cheng
claimed_verified
2024-11-15T09:30:04.697Z
65084ce79db6e2495cc52cc6
https://cdn-avatars.huggingface.co/v1/production/uploads/65084ce79db6e2495cc52cc6/MbwRARtrDfjVi_DHZW6-r.jpeg
Tianhao Cheng
user
crazycth
672d8d720028ad1c50f6c7d4
Jason Klein Liu
claimed_verified
2024-11-16T19:47:05.609Z
6731c04d5f55903a1d8c307c
/avatars/704b1d628c55e3141194b08736f21267.svg
Jason Klein Liu
user
jasonkleinlove
672d8d720028ad1c50f6c7d5
Jiaran Hao
672d8d720028ad1c50f6c7d6
Liuyihan Song
672d8d720028ad1c50f6c7d7
Yang Xu
672d8d720028ad1c50f6c7d8
J. Yang
672d8d720028ad1c50f6c7d9
J. H. Liu
672d8d720028ad1c50f6c7da
Chenchen Zhang
672d8d720028ad1c50f6c7db
Linzheng Chai
claimed_verified
2024-11-15T09:30:06.532Z
64ba096e760936217a3ad2e2
https://cdn-avatars.huggingface.co/v1/production/uploads/64ba096e760936217a3ad2e2/aNQK83Jg5PsBkY0UDg-RA.jpeg
Linzheng Chai
user
Challenging666
672d8d720028ad1c50f6c7dc
Ruifeng Yuan
672d8d720028ad1c50f6c7dd
Zhaoxiang Zhang
672d8d720028ad1c50f6c7de
Jie Fu
claimed_verified
2024-11-15T09:30:11.161Z
641a6895fb5ffff5ac79d593
https://cdn-avatars.huggingface.co/v1/production/uploads/641a6895fb5ffff5ac79d593/vxvwsto3llOEWGqQKGMYx.jpeg
Jie Fu
user
bigaidream
672d8d720028ad1c50f6c7df
Qian Liu
claimed_verified
2024-11-15T09:30:55.612Z
612ee6a7b960e78c6d2319d4
https://cdn-avatars.huggingface.co/v1/production/uploads/612ee6a7b960e78c6d2319d4/2Hu9BaAyXbyh1vt0v1Qui.jpeg
Qian Liu
user
SivilTaram
672d8d720028ad1c50f6c7e0
Ge Zhang
claimed_verified
2024-11-15T09:31:20.477Z
638efcf4c67af472d316d424
/avatars/97a57859d7d87a3a8f1bb41d32a72bc2.svg
Ge Zhang
user
zhangysk
672d8d720028ad1c50f6c7e1
Zili Wang
extracted_pending
2024-11-08T04:03:02.446Z
632c59bbea6e62428ab3b480
https://cdn-avatars.huggingface.co/v1/production/uploads/632c59bbea6e62428ab3b480/XroXmPnGl1fDrSnN1bQ99.jpeg
Zili Wang
user
Zenithwang
672d8d720028ad1c50f6c7e2
Yuan Qi
672d8d720028ad1c50f6c7e3
Yinghui Xu
672d8d720028ad1c50f6c7e4
Wei Chu
OpenCoder: The Open Cookbook for Top-Tier Code Large Language Models
Large language models (LLMs) for code have become indispensable in various
domains, including code generation, reasoning tasks and agent systems.While
open-access code LLMs are increasingly approaching the performance levels of
proprietary models, high-quality code LLMs suitable for rigorous scientific
investigation, particularly those with reproducible data processing pipelines
and transparent training protocols, remain limited. The scarcity is due to
various challenges, including resource constraints, ethical considerations, and
the competitive advantages of keeping models advanced. To address the gap, we
introduce OpenCoder, a top-tier code LLM that not only achieves performance
comparable to leading models but also serves as an ``open cookbook'' for the
research community. Unlike most prior efforts, we release not only model
weights and inference code, but also the reproducible training data, complete
data processing pipeline, rigorous experimental ablation results, and detailed
training protocols for open scientific research. Through this comprehensive
release, we identify the key ingredients for building a top-tier code LLM: (1)
code optimized heuristic rules for data cleaning and methods for data
deduplication, (2) recall of text corpus related to code and (3) high-quality
synthetic data in both annealing and supervised fine-tuning stages. By offering
this level of openness, we aim to broaden access to all aspects of a top-tier
code LLM, with OpenCoder serving as both a powerful model and an open
foundation to accelerate research, and enable reproducible advancements in code
AI.
OpenCoder is a top-tier open-access code LLM that provides comprehensive reproducible resources, including data, pipelines, and protocols, to advance code AI research.
large language models code generation reasoning tasks agent systems open-print code LLMs proprietary models high-quality code LLMs reproducible data processing pipelines transparent training protocols resource constraints ethical considerations OpenCoder model weights inference code experimental ablation results detailed training protocols data cleaning data deduplication text corpus related to code high-quality synthetic data annealing supervised fine-tuning