OpenELM: An Efficient Language Model Family with Open-source Training
  and Inference Framework
The reproducibility and transparency of large language models are crucial for
advancing open research, ensuring the trustworthiness of results, and enabling
investigations into data and model biases, as well as potential risks. To this
end, we release OpenELM, a state-of-the-art open language model. OpenELM uses a
layer-wise scaling strategy to efficiently allocate parameters within each
layer of the transformer model, leading to enhanced accuracy. For example, with
a parameter budget of approximately one billion parameters, OpenELM exhibits a
2.36% improvement in accuracy compared to OLMo while requiring 2times fewer
pre-training tokens.
  Diverging from prior practices that only provide model weights and inference
code, and pre-train on private datasets, our release includes the complete
framework for training and evaluation of the language model on publicly
available datasets, including training logs, multiple checkpoints, and
pre-training configurations. We also release code to convert models to MLX
library for inference and fine-tuning on Apple devices. This comprehensive
release aims to empower and strengthen the open research community, paving the
way for future open research endeavors.
  Our source code along with pre-trained model weights and training recipes is
available at https://github.com/apple/corenet. Additionally, \model
models can be found on HuggingFace at:
https://huggingface.co/apple/OpenELM.
662858d84a0dc08679e8f359
Sachin Mehta
admin_assigned
2024-04-24T11:15:46.478Z
64f792652003abb61bf5c1b8
/avatars/9721810de3519ece50b059c154baa9ca.svg
Sachin Mehta
user
sacmehta
662858d84a0dc08679e8f35a
Mohammad Hossein Sekhavat
admin_assigned
2024-04-24T11:15:53.385Z
65ca91986938a6a81fd80d02
/avatars/efac7945ac1df4e8949e77465ef0dcbd.svg
Mohammad Hossein Sekhavat
user
msekhavat
662858d84a0dc08679e8f35b
Qingqing Cao
claimed_verified
2024-04-24T11:11:23.174Z
5f07d37219cb630495b81306
https://cdn-avatars.huggingface.co/v1/production/uploads/1595440980042-5f07d37219cb630495b81306.jpeg
Qingqing Cao
user
csarron
662858d84a0dc08679e8f35c
Maxwell Horton
admin_assigned
2024-04-24T10:39:58.055Z
647a5afc732a0cd866eb512f
/avatars/fbd0ba2a7ba70414ac903e583ce62933.svg
Maxwell Horton
user
mchorton
662858d84a0dc08679e8f35d
Yanzi Jin
admin_assigned
2024-04-24T11:15:59.745Z
6627307fa39494ebb3a39194
/avatars/99290642fd6d4a6162f52cfe5d46c648.svg
Yanzi Jin
user
yjin25
662858d84a0dc08679e8f35e
Chenfan Sun
claimed_verified
2024-10-17T11:30:02.341Z
658895a1c4b20046639a7e59
/avatars/f80c3f58efa5d22464590bee7a6041a4.svg
Chenfan Sun
user
erpang007
662858d84a0dc08679e8f35f
Iman Mirzadeh
admin_assigned
2024-04-24T11:16:15.272Z
66280a4c7c988f33b795f90f
/avatars/7a618b39c4ed20f5490ea64fbb67b5ac.svg
Iman Mirzadeh
user
imirzadeh-apple
662858d84a0dc08679e8f360
Mahyar Najibi
admin_assigned
2024-04-24T11:16:33.449Z
661d992d69a65c0fb0717013
/avatars/9d1f20baff742de90a50232db54d2407.svg
Mahyar Najibi
user
mahyar-najibi
662858d84a0dc08679e8f361
Dmitry Belenko
claimed_verified
2024-04-24T11:11:25.057Z
63185e1a70b013d6609416ba
https://cdn-avatars.huggingface.co/v1/production/uploads/63185e1a70b013d6609416ba/LOB6bJ71wwgamgN3MCuFb.jpeg
Dmitry Belenko
user
depthwise
662858d84a0dc08679e8f362
Peter Zatloukal
662858d84a0dc08679e8f363
Mohammad Rastegari
admin_assigned
2024-04-24T11:16:48.273Z
65d4c7734c2163d05eab4110
/avatars/b116a9aca7b0a2a4836a29cb4f069d42.svg
Mohammad Rastegari
user
mrastegari
OpenELM: An Efficient Language Model Family with Open-source Training
  and Inference Framework
The reproducibility and transparency of large language models are crucial for
advancing open research, ensuring the trustworthiness of results, and enabling
investigations into data and model biases, as well as potential risks. To this
end, we release OpenELM, a state-of-the-art open language model. OpenELM uses a
layer-wise scaling strategy to efficiently allocate parameters within each
layer of the transformer model, leading to enhanced accuracy. For example, with
a parameter budget of approximately one billion parameters, OpenELM exhibits a
2.36% improvement in accuracy compared to OLMo while requiring 2times fewer
pre-training tokens.
  Diverging from prior practices that only provide model weights and inference
code, and pre-train on private datasets, our release includes the complete
framework for training and evaluation of the language model on publicly
available datasets, including training logs, multiple checkpoints, and
pre-training configurations. We also release code to convert models to MLX
library for inference and fine-tuning on Apple devices. This comprehensive
release aims to empower and strengthen the open research community, paving the
way for future open research endeavors.
  Our source code along with pre-trained model weights and training recipes is
available at https://github.com/apple/corenet. Additionally, \model
models can be found on HuggingFace at:
https://huggingface.co/apple/OpenELM.
OpenELM, an open language model, uses layer-wise scaling to enhance accuracy and is released with comprehensive training and evaluation frameworks, enabling transparent and reproducible research.
layer-wise scaling transformer model OpenELM OLMo pre-training tokens training logs checkpoints pre-training configurations MLX library