StarCoder: may the source be with you!
The BigCode community, an open-scientific collaboration working on the
responsible development of Large Language Models for Code (Code LLMs),
introduces StarCoder and StarCoderBase: 15.5B parameter models with 8K context
length, infilling capabilities and fast large-batch inference enabled by
multi-query attention. StarCoderBase is trained on 1 trillion tokens sourced
from The Stack, a large collection of permissively licensed GitHub repositories
with inspection tools and an opt-out process. We fine-tuned StarCoderBase on
35B Python tokens, resulting in the creation of StarCoder. We perform the most
comprehensive evaluation of Code LLMs to date and show that StarCoderBase
outperforms every open Code LLM that supports multiple programming languages
and matches or outperforms the OpenAI code-cushman-001 model. Furthermore,
StarCoder outperforms every model that is fine-tuned on Python, can be prompted
to achieve 40\% pass@1 on HumanEval, and still retains its performance on other
programming languages. We take several important steps towards a safe
open-access model release, including an improved PII redaction pipeline and a
novel attribution tracing tool, and make the StarCoder models publicly
available under a more commercially viable version of the Open Responsible AI
Model license.
645c4c127848314a460d1759
Raymond Li
645c4c127848314a460d175a
Loubna Ben Allal
admin_assigned
2023-05-11T08:27:56.619Z
61c141342aac764ce1654e43
https://cdn-avatars.huggingface.co/v1/production/uploads/61c141342aac764ce1654e43/81AwoT5IQ_Xdw0OVw7TKu.jpeg
Loubna Ben Allal
user
loubnabnl
645c4c127848314a460d175b
Yangtian Zi
claimed_verified
2023-05-15T07:34:40.656Z
6354a759eca5a823e3bc89ba
/avatars/e4ad8a56bf416ab11a1cfaf64f2edc09.svg
Yangtian Zi
user
ytzi
645c4c127848314a460d175c
Niklas Muennighoff
admin_assigned
2023-05-11T08:29:15.774Z
5f1eb362eec0ad2a071ad6e2
https://cdn-avatars.huggingface.co/v1/production/uploads/5f1eb362eec0ad2a071ad6e2/IXMYkYKuTwn6kBdWnQeeY.png
Niklas Muennighoff
user
Muennighoff
645c4c127848314a460d175d
Denis Kocetkov
645c4c127848314a460d175e
Chenghao Mou
645c4c127848314a460d175f
Marc Marone
claimed_verified
2023-05-11T15:00:49.640Z
63e410e88083f19a218be964
/avatars/ce271d1686f6e5ee1f5b2d429cb60de6.svg
Marc Marone
user
mmarone
645c4c127848314a460d1760
Christopher Akiki
admin_assigned
2023-05-11T08:28:12.968Z
5e70f6048ce3c604d78fe133
https://cdn-avatars.huggingface.co/v1/production/uploads/1646492542174-5e70f6048ce3c604d78fe133.jpeg
Christopher Akiki
user
christopher
645c4c127848314a460d1761
Jia Li
645c4c127848314a460d1762
Jenny Chim
claimed_verified
2023-05-15T15:28:34.958Z
5fc8e6e9628d967e353416b3
/avatars/d913dff9380e711aa4ecf368e3070f88.svg
Jenny Chim
user
j-chim
645c4c127848314a460d1763
Qian Liu
claimed_verified
2023-05-11T08:25:20.910Z
612ee6a7b960e78c6d2319d4
https://cdn-avatars.huggingface.co/v1/production/uploads/612ee6a7b960e78c6d2319d4/2Hu9BaAyXbyh1vt0v1Qui.jpeg
Qian Liu
user
SivilTaram
645c4c127848314a460d1764
Evgenii Zheltonozhskii
claimed_verified
2023-05-11T15:00:36.990Z
61ae250602b8655e3c8973cd
https://cdn-avatars.huggingface.co/v1/production/uploads/61ae250602b8655e3c8973cd/aXp22-B-oKlm_h6kLx9Uv.jpeg
Evgenii Zheltonozhskii
user
EvgeniyZh
645c4c127848314a460d1765
Terry Yue Zhuo
claimed_verified
2023-05-15T07:34:57.914Z
62b7fb545233925f253531c8
https://cdn-avatars.huggingface.co/v1/production/uploads/62b7fb545233925f253531c8/W50u2G1HK3EtUKHRU189V.jpeg
Terry Yue Zhuo
user
terryyz
645c4c127848314a460d1766
Thomas Wang
admin_assigned
2023-05-11T08:28:52.145Z
61140e35d08630d2676c9829
/avatars/3d6e4b4d02eda7b5ef28e1cc0fb8e08a.svg
Thomas Wang
user
TimeRobber
645c4c127848314a460d1767
Olivier Dehaene
admin_assigned
2023-05-11T08:28:32.574Z
62a093d63e7d1dda047039fc
https://cdn-avatars.huggingface.co/v1/production/uploads/62a093d63e7d1dda047039fc/QGpVSKuJLwl2EsiffCYML.jpeg
Olivier Dehaene
user
olivierdehaene
645c4c127848314a460d1768
Mishig Davaadorj
admin_assigned
2023-05-11T08:34:42.462Z
60a551a34ecc5d054c8ad93e
https://cdn-avatars.huggingface.co/v1/production/uploads/60a551a34ecc5d054c8ad93e/dhcBFtwNLcKqqASxniyVw.jpeg
Mishig Davaadorj
user
mishig
645c4c127848314a460d1769
Joel Lamy-Poirier
admin_assigned
2023-05-12T08:37:39.601Z
6372ec0e759c1ca82045c43d
/avatars/86d5dbdb10a0cbf9a39a32e2d87361b6.svg
Joel Lamy-Poirier
user
jlamypoirier
645c4c127848314a460d176a
João Monteiro
claimed_verified
2023-05-11T15:00:27.548Z
63347fed495073b88704cf02
/avatars/cb5545adaf1cfe0166119c586f4ac9c8.svg
João Monteiro
user
joaomonteirof
645c4c127848314a460d176b
Oleh Shliazhko
claimed_verified
2023-05-11T15:00:46.540Z
630e7fb01ef92d4e37a56400
/avatars/418a19c9a0deaba6ee4938e5c941253e.svg
Oleh Shliazhko
user
olmer
645c4c127848314a460d176c
Nicolas Gontier
645c4c127848314a460d176d
Nicholas Meade
645c4c127848314a460d176e
Armel Zebaze
645c4c127848314a460d176f
Ming-Ho Yee
645c4c127848314a460d1770
Logesh Kumar Umapathi
claimed_verified
2023-05-11T15:00:52.569Z
6052dd84a7226b25aaeea2be
/avatars/d130e7051d81303130883709aacb2618.svg
Logesh Kumar umapathi
user
infinitylogesh
645c4c127848314a460d1771
Jian Zhu
645c4c127848314a460d1772
Benjamin Lipkin
claimed_verified
2023-05-11T15:00:56.047Z
613a6efea488863c1920731e
https://cdn-avatars.huggingface.co/v1/production/uploads/1664283416127-613a6efea488863c1920731e.jpeg
Ben Lipkin
user
benlipkin
645c4c127848314a460d1773
Muhtasham Oblokulov
claimed_verified
2023-05-11T15:00:00.456Z
5fcaabed246881afd5b00167
https://cdn-avatars.huggingface.co/v1/production/uploads/1651847561574-5fcaabed246881afd5b00167.jpeg
Muhtasham Oblokulov
user
muhtasham
645c4c127848314a460d1774
Zhiruo Wang
645c4c127848314a460d1775
Rudra Murthy
645c4c127848314a460d1776
Jason Stillerman
claimed_verified
2023-05-15T07:34:52.595Z
6178d842267320abb99df776
/avatars/8739705ef7eabc945a1ef3c07196af13.svg
Jason Stillerman
user
stillerman
645c4c127848314a460d1777
Siva Sankalp Patel
claimed_verified
2023-05-12T08:11:30.592Z
618c2037526de8dfed6b3180
https://cdn-avatars.huggingface.co/v1/production/uploads/618c2037526de8dfed6b3180/QhobNWuHwVzyG9OtntzSp.jpeg
Siva Sankalp
user
sivasankalpp
645c4c127848314a460d1778
Dmitry Abulkhanov
claimed_verified
2023-05-19T10:28:18.648Z
636a9a07e3ad78bc68b1a5a2
https://cdn-avatars.huggingface.co/v1/production/uploads/1668020490988-636a9a07e3ad78bc68b1a5a2.jpeg
Dmitry Abulkhanov
user
mponty
645c4c127848314a460d1779
Marco Zocca
claimed_verified
2024-06-30T20:59:21.536Z
6282a048e84c35bcb7d90a76
https://cdn-avatars.huggingface.co/v1/production/uploads/1665103679595-6282a048e84c35bcb7d90a76.jpeg
Marco Zocca
user
ocramz
645c4c127848314a460d177a
Manan Dey
claimed_verified
2023-05-11T15:00:24.018Z
5fc53305fa6eef7667a4d691
https://cdn-avatars.huggingface.co/v1/production/uploads/1623089209793-5fc53305fa6eef7667a4d691.jpeg
Manan Dey
user
manandey
645c4c127848314a460d177b
Zhihan Zhang
claimed_verified
2023-05-12T08:11:22.127Z
63bf9695da08ed054400205e
/avatars/5223b212431337c6186db96ac70b7ae7.svg
Zhihan Zhang
user
zhihz0535
645c4c127848314a460d177c
Nour Fahmy
645c4c127848314a460d177d
Urvashi Bhattacharyya
645c4c127848314a460d177e
Wenhao Yu
claimed_verified
2023-05-11T19:01:44.456Z
5feab3a28a3201f8e554c969
https://cdn-avatars.huggingface.co/v1/production/uploads/1660795228685-5feab3a28a3201f8e554c969.png
Wenhao Yu
user
wyu1
645c4c127848314a460d177f
Swayam Singh
claimed_verified
2023-05-11T08:25:18.334Z
62f47c093561a52aa5a67c90
https://cdn-avatars.huggingface.co/v1/production/uploads/62f47c093561a52aa5a67c90/d4sFnllrLH5BWbZDRNvMn.jpeg
Swayam Singh
user
rootacess
645c4c127848314a460d1780
Sasha Luccioni
admin_assigned
2023-05-11T08:34:28.553Z
60edd0133e2c73a9a21455f5
https://cdn-avatars.huggingface.co/v1/production/uploads/60edd0133e2c73a9a21455f5/yK1G-Fv-YjYb7v_chkz3p.jpeg
Sasha Luccioni
user
sasha
645c4c127848314a460d1781
Paulo Villegas
claimed_verified
2023-05-15T07:34:55.173Z
60d433541af6803389820846
https://cdn-avatars.huggingface.co/v1/production/uploads/1626374472249-60d433541af6803389820846.jpeg
Paulo Villegas
user
Paulo
645c4c127848314a460d1782
Maxim Kunakov
645c4c127848314a460d1783
Fedor Zhdanov
645c4c127848314a460d1784
Manuel Romero
admin_assigned
2023-05-11T08:30:10.779Z
5e4318d616b09a31220980d6
https://cdn-avatars.huggingface.co/v1/production/uploads/5e4318d616b09a31220980d6/24rMJ_vPh3gW9ZEmj64xr.png
Manuel Romero
user
mrm8488
645c4c127848314a460d1785
Tony Lee
645c4c127848314a460d1786
Nadav Timor
claimed_verified
2023-05-12T08:11:16.640Z
62be5fc0c990a0a515da5f34
https://cdn-avatars.huggingface.co/v1/production/uploads/1674221847840-62be5fc0c990a0a515da5f34.jpeg
Nadav Timor
user
Nadav-Timor
645c4c127848314a460d1787
Jennifer Ding
claimed_verified
2024-09-06T09:05:59.294Z
62a0da842e30aaf94ebaaa12
https://cdn-avatars.huggingface.co/v1/production/uploads/1679073278459-62a0da842e30aaf94ebaaa12.jpeg
Jennifer Ding
user
jending12
645c4c127848314a460d1788
Claire Schlesinger
claimed_verified
2023-05-15T07:34:49.659Z
63d4aa23ff1384ce6c683544
https://cdn-avatars.huggingface.co/v1/production/uploads/63d4aa23ff1384ce6c683544/7RQIh5dyak8wanCdS1dGu.jpeg
Claire Schlesinger
user
esslushy
645c4c127848314a460d1789
Hailey Schoelkopf
645c4c127848314a460d178a
Jan Ebert
claimed_verified
2023-05-11T15:00:40.385Z
630dc2eda16f7b175da838dd
/avatars/11f515923b3d378c056d99ff7e52d6aa.svg
Jan
user
jaeb
645c4c127848314a460d178b
Tri Dao
admin_assigned
2023-08-14T15:33:40.468Z
64b8a6b5cf14c2fabe98159b
/avatars/dbc009451865435bf290791beadc4723.svg
Tri Dao
user
tridao
645c4c127848314a460d178c
Mayank Mishra
claimed_verified
2023-05-11T15:00:34.183Z
62cd5057674cdb524450093d
https://cdn-avatars.huggingface.co/v1/production/uploads/62cd5057674cdb524450093d/f67rlrdsKPRTLdXCXoa_X.jpeg
Mayank Mishra
user
mayank-mishra
645c4c127848314a460d178d
Alex Gu
claimed_verified
2024-03-19T08:54:48.209Z
6179abf8ec6ce4dc2e5f2376
/avatars/d10c6a1b350146b36949a24220471295.svg
Alex Gu
user
minimario
645c4c127848314a460d178e
Jennifer Robinson
645c4c127848314a460d178f
Carolyn Jane Anderson
645c4c127848314a460d1790
Brendan Dolan-Gavitt
645c4c127848314a460d1791
Danish Contractor
claimed_verified
2025-06-11T08:41:53.965Z
607a8623921db717010c7cae
/avatars/37f5263dbfb6294ac885eef1ea9db5d2.svg
danish
user
danish
645c4c127848314a460d1792
Siva Reddy
645c4c127848314a460d1793
Daniel Fried
claimed_verified
2023-05-11T19:01:47.280Z
623e041cdcda6a7153052d02
https://cdn-avatars.huggingface.co/v1/production/uploads/623e041cdcda6a7153052d02/dCAe49oOZDsyp40xqiB-L.png
Daniel Fried
user
dpfried
645c4c127848314a460d1794
Dzmitry Bahdanau
645c4c127848314a460d1795
Yacine Jernite
admin_assigned
2023-05-11T08:33:52.890Z
5ee3a7cd2a3eae3cbdad1305
https://cdn-avatars.huggingface.co/v1/production/uploads/1594144055859-5ee3a7cd2a3eae3cbdad1305.jpeg
Yacine Jernite
user
yjernite
645c4c127848314a460d1796
Carlos Muñoz Ferrandis
admin_assigned
2023-05-11T08:29:34.163Z
629f05bddaca51e26a7cfef1
https://cdn-avatars.huggingface.co/v1/production/uploads/1654588848873-noauth.jpeg
Carlos Muñoz Ferrandis
user
CarlosMF
645c4c127848314a460d1797
Sean Hughes
claimed_verified
2023-05-11T19:01:36.853Z
633dd1e467e67df066ef2cbf
/avatars/121fff6aefd09565fd2b01d7fcda6757.svg
Sean Hughes
user
hughesthe1st
645c4c127848314a460d1798
Thomas Wolf
admin_assigned
2023-05-11T08:35:06.520Z
5df7e9e5da6d0311fd3d53f9
https://cdn-avatars.huggingface.co/v1/production/uploads/1583857746553-5df7e9e5da6d0311fd3d53f9.jpeg
Thomas Wolf
user
thomwolf
645c4c127848314a460d1799
Arjun Guha
claimed_verified
2023-05-11T15:00:31.316Z
62d8315bad693a1a962864b3
https://cdn-avatars.huggingface.co/v1/production/uploads/1664332914111-62d8315bad693a1a962864b3.png
Arjun Guha
user
arjunguha
645c4c127848314a460d179a
Leandro von Werra
claimed_verified
2023-05-11T08:25:14.980Z
5e48005437cb5b49818287a5
https://cdn-avatars.huggingface.co/v1/production/uploads/5e48005437cb5b49818287a5/4uCXGGui-9QifAT4qelxU.png
Leandro von Werra
user
lvwerra
645c4c127848314a460d179b
Harm de Vries
admin_assigned
2023-05-12T08:37:03.497Z
62ce91e8b5811ba53c0ce7b9
https://cdn-avatars.huggingface.co/v1/production/uploads/62ce91e8b5811ba53c0ce7b9/ZNgd-p0h-q1j374QqEGrb.jpeg
Harm de Vries
user
harmdevries
StarCoder: may the source be with you!
The BigCode community, an open-scientific collaboration working on the
responsible development of Large Language Models for Code (Code LLMs),
introduces StarCoder and StarCoderBase: 15.5B parameter models with 8K context
length, infilling capabilities and fast large-batch inference enabled by
multi-query attention. StarCoderBase is trained on 1 trillion tokens sourced
from The Stack, a large collection of permissively licensed GitHub repositories
with inspection tools and an opt-out process. We fine-tuned StarCoderBase on
35B Python tokens, resulting in the creation of StarCoder. We perform the most
comprehensive evaluation of Code LLMs to date and show that StarCoderBase
outperforms every open Code LLM that supports multiple programming languages
and matches or outperforms the OpenAI code-cushman-001 model. Furthermore,
StarCoder outperforms every model that is fine-tuned on Python, can be prompted
to achieve 40\% pass@1 on HumanEval, and still retains its performance on other
programming languages. We take several important steps towards a safe
open-access model release, including an improved PII redaction pipeline and a
novel attribution tracing tool, and make the StarCoder models publicly
available under a more commercially viable version of the Open Responsible AI
Model license.
StarCoder, a 15.5B parameter LLM trained on 1 trillion tokens, outperforms other open Code LLMs across multiple languages and fine-tuned Python, with safety enhancements and publicly available under the Open Responsible AI Model license.
Large Language Models Code LLMs StarCoder StarCoderBase 8K context length infilling capabilities multi-query attention The Stack Python tokens HumanEval PII redaction attribution tracing tool Open Responsible AI Model license