LLM360: Towards Fully Transparent Open-Source LLMs
The recent surge in open-source Large Language Models (LLMs), such as LLaMA,
Falcon, and Mistral, provides diverse options for AI practitioners and
researchers. However, most LLMs have only released partial artifacts, such as
the final model weights or inference code, and technical reports increasingly
limit their scope to high-level design choices and surface statistics. These
choices hinder progress in the field by degrading transparency into the
training of LLMs and forcing teams to rediscover many details in the training
process. We present LLM360, an initiative to fully open-source LLMs, which
advocates for all training code and data, model checkpoints, and intermediate
results to be made available to the community. The goal of LLM360 is to support
open and collaborative AI research by making the end-to-end LLM training
process transparent and reproducible by everyone. As a first step of LLM360, we
release two 7B parameter LLMs pre-trained from scratch, Amber and CrystalCoder,
including their training code, data, intermediate checkpoints, and analyses (at
https://www.llm360.ai). We are committed to continually pushing the boundaries
of LLMs through this open-source effort. More large-scale and stronger models
are underway and will be released in the future.
6577f192197a6182c338a64f
Zhengzhong Liu
admin_assigned
2023-12-12T08:28:37.331Z
62fbdc67c776fd8821ae3f2d
https://cdn-avatars.huggingface.co/v1/production/uploads/62fbdc67c776fd8821ae3f2d/cI7iAZOL40RUYluo5ZVTU.png
Zhengzhong Liu
user
hunterhector
6577f192197a6182c338a650
Aurick Qiao
admin_assigned
2023-12-12T08:28:43.735Z
630e8f0bf6f6d700f50ebd2e
https://cdn-avatars.huggingface.co/v1/production/uploads/630e8f0bf6f6d700f50ebd2e/ELJAinK1XpVJ8aslZ3BIW.jpeg
Aurick Qiao
user
aurick
6577f192197a6182c338a651
Willie Neiswanger
claimed_verified
2023-12-12T08:17:57.765Z
644bf65522d211df6444a7f4
https://cdn-avatars.huggingface.co/v1/production/uploads/644bf65522d211df6444a7f4/k_ddZdQDg2fzhwjI1EXyx.jpeg
Willie Neiswanger
user
willieneis
6577f192197a6182c338a652
Hongyi Wang
admin_assigned
2023-12-12T08:29:20.507Z
64e5414fe56d371e42da69c4
/avatars/7d4a101b0338410bc736e04d7a611aca.svg
Hongyi Wang
user
hwang595
6577f192197a6182c338a653
Bowen Tan
admin_assigned
2023-12-12T08:29:35.182Z
64c52182d43e4dee5177e8b7
/avatars/d13edf5fe3f7f21bb2f22363378b0d9d.svg
Bowen Tan
user
btan2
6577f192197a6182c338a654
Tianhua Tao
admin_assigned
2023-12-12T08:29:41.752Z
61ecb698382b3b92fea7305f
/avatars/b1bcf0d9ade12172810cb65dc9c4e4d8.svg
Tianhua Tao
user
Tianhua
6577f192197a6182c338a655
Junbo Li
admin_assigned
2023-12-12T08:29:48.130Z
6317d681743def9b3612ebeb
/avatars/9940b35d5f8ae5df417bb020b609a8d0.svg
Junbo Li
user
ljb121002
6577f192197a6182c338a656
Yuqi Wang
6577f192197a6182c338a657
Suqi Sun
6577f192197a6182c338a658
Omkar Pangarkar
admin_assigned
2023-12-12T08:30:39.543Z
644086c17663594a126575dd
/avatars/ba8f2f87bbc3ee7b4da444a4b5097e4c.svg
Omkar Pangarkar
user
omkarenator
6577f192197a6182c338a659
Richard Fan
6577f192197a6182c338a65a
Yi Gu
claimed_verified
2024-06-18T07:21:17.481Z
64c2ab94e818eec6128bfeb7
/avatars/459794e67d43a4b739ddcf90d1921b31.svg
Yi Gu
user
wuqing157
6577f192197a6182c338a65b
Victor Miller
admin_assigned
2023-12-12T08:31:27.772Z
638a93f1ed88cf97afd53e42
/avatars/147ed42f13847b5e4d534511ef5388a3.svg
Victor Miller
user
victormiller
6577f192197a6182c338a65c
Yonghao Zhuang
admin_assigned
2023-12-12T08:31:35.156Z
64ab987073790912c7ad79e6
/avatars/bfa00418c46da0fd1e522d6561944ec9.svg
Yonghao Zhuang
user
ZYHowell
6577f192197a6182c338a65d
Guowei He
6577f192197a6182c338a65e
Haonan Li
admin_assigned
2023-12-12T08:32:05.099Z
643934ecb9ac1d55f41cefd7
https://cdn-avatars.huggingface.co/v1/production/uploads/643934ecb9ac1d55f41cefd7/v41FZ2Ar6nVXEkf-w5axf.jpeg
Haonan Li
user
lmlmcat
6577f192197a6182c338a65f
Fajri Koto
admin_assigned
2023-12-12T08:32:15.632Z
643fb246c2ec31af16aa9313
https://cdn-avatars.huggingface.co/v1/production/uploads/643fb246c2ec31af16aa9313/zCDYOVGdIrncX-k9o7bM6.png
Fajri Koto
user
fajrikoto
6577f192197a6182c338a660
Liping Tang
admin_assigned
2023-12-12T08:32:23.281Z
61a05a0c17ff4c17bba2200f
https://cdn-avatars.huggingface.co/v1/production/uploads/61a05a0c17ff4c17bba2200f/uMGfYrYubVpmgWXytNHAj.png
Liping Tang
user
Liping
6577f192197a6182c338a661
Nikhil Ranjan
admin_assigned
2023-12-12T08:32:47.304Z
64de07f2228f31beb858d458
/avatars/0dfdf90fbd871221466497dc49f6e356.svg
Nikhil Ranjan
user
nikhilranjan
6577f192197a6182c338a662
Zhiqiang Shen
admin_assigned
2023-12-12T08:32:53.455Z
649d5a3f4e08d59d20014652
/avatars/017eccd6aec02137c1beebfe4c951720.svg
Zhiqiang Shen
user
Jason0214
6577f192197a6182c338a663
Xuguang Ren
6577f192197a6182c338a664
Roberto Iriondo
admin_assigned
2023-12-12T08:33:13.071Z
652417ebf172523dc3d57eb4
/avatars/53313181df859b3e7c5eab75211327dd.svg
Roberto Iriondo
user
roberto12345
6577f192197a6182c338a665
Cun Mu
admin_assigned
2023-12-12T08:33:20.064Z
64f3fc811da608d0842bca3f
/avatars/b2cd0eae464f577202afd982cd49bd2b.svg
Cun Mu 
user
mudongdong
6577f192197a6182c338a666
Zhiting Hu
6577f192197a6182c338a667
Mark Schulze
6577f192197a6182c338a668
Preslav Nakov
admin_assigned
2023-12-12T08:33:39.480Z
647f7eb25e1bc4753746bf9f
/avatars/cc9c6210fdc822d8a106937e747dff41.svg
Preslav Nakov
user
preslavnakov
6577f192197a6182c338a669
Tim Baldwin
claimed_verified
2024-02-23T13:02:28.152Z
65d895d8d435dce78392065f
/avatars/68c7185d3dd1adf78927b91849978d62.svg
Tim Baldwin
user
tbaldwin
6577f192197a6182c338a66a
Eric P. Xing
admin_assigned
2023-12-12T08:33:54.472Z
64ff67722ad36636be6c4542
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/sLIrNelAWPVOy4e3oo5LB.jpeg
Eric Xing
user
EricX003
LLM360: Towards Fully Transparent Open-Source LLMs
The recent surge in open-source Large Language Models (LLMs), such as LLaMA,
Falcon, and Mistral, provides diverse options for AI practitioners and
researchers. However, most LLMs have only released partial artifacts, such as
the final model weights or inference code, and technical reports increasingly
limit their scope to high-level design choices and surface statistics. These
choices hinder progress in the field by degrading transparency into the
training of LLMs and forcing teams to rediscover many details in the training
process. We present LLM360, an initiative to fully open-source LLMs, which
advocates for all training code and data, model checkpoints, and intermediate
results to be made available to the community. The goal of LLM360 is to support
open and collaborative AI research by making the end-to-end LLM training
process transparent and reproducible by everyone. As a first step of LLM360, we
release two 7B parameter LLMs pre-trained from scratch, Amber and CrystalCoder,
including their training code, data, intermediate checkpoints, and analyses (at
https://www.llm360.ai). We are committed to continually pushing the boundaries
of LLMs through this open-source effort. More large-scale and stronger models
are underway and will be released in the future.
LLM360 initiative promotes full transparency and reproducibility in LLM training by open-sourcing training code, data, model checkpoints, and intermediate results.
Large Language Models LLaMA Falcon Mistral training code data model checkpoints intermediate results pre-trained from scratch Amber CrystalCoder