OLMoE: Open Mixture-of-Experts Language Models
We introduce OLMoE, a fully open, state-of-the-art language model leveraging
sparse Mixture-of-Experts (MoE). OLMoE-1B-7B has 7 billion (B) parameters but
uses only 1B per input token. We pretrain it on 5 trillion tokens and further
adapt it to create OLMoE-1B-7B-Instruct. Our models outperform all available
models with similar active parameters, even surpassing larger ones like
Llama2-13B-Chat and DeepSeekMoE-16B. We present various experiments on MoE
training, analyze routing in our model showing high specialization, and
open-source all aspects of our work: model weights, training data, code, and
logs.
66d7c90fe1aee6b7145bd5a3
Niklas Muennighoff
extracted_confirmed
2024-09-04T02:49:16.535Z
5f1eb362eec0ad2a071ad6e2
https://cdn-avatars.huggingface.co/v1/production/uploads/5f1eb362eec0ad2a071ad6e2/IXMYkYKuTwn6kBdWnQeeY.png
Niklas Muennighoff
user
Muennighoff
66d7c90fe1aee6b7145bd5a4
Luca Soldaini
admin_assigned
2024-09-04T08:04:06.994Z
5f04d8c45d08220171a0ad32
https://cdn-avatars.huggingface.co/v1/production/uploads/5f04d8c45d08220171a0ad32/uXEta6nqBabrUlAOXnS5g.jpeg
Luca Soldaini
user
soldni
66d7c90fe1aee6b7145bd5a5
Dirk Groeneveld
admin_assigned
2024-09-04T08:04:13.711Z
60369745413a78f892e7339c
https://cdn-avatars.huggingface.co/v1/production/uploads/1636671879171-60369745413a78f892e7339c.png
Dirk Groeneveld
user
dirkgr
66d7c90fe1aee6b7145bd5a6
Kyle Lo
66d7c90fe1aee6b7145bd5a7
Jacob Morrison
admin_assigned
2024-09-04T08:05:12.456Z
63f51f6cb51da4d61da72937
/avatars/6023ecaafb28b7599f59816381d58bd0.svg
Jacob Morrison
user
jacobmorrison
66d7c90fe1aee6b7145bd5a8
Sewon Min
admin_assigned
2024-09-04T08:05:30.534Z
63a76d0de27a6dbd485fe863
https://cdn-avatars.huggingface.co/v1/production/uploads/63a76d0de27a6dbd485fe863/qJJwHOuvyQGq1o0KscOF_.jpeg
Sewon Min
user
sewon
66d7c90fe1aee6b7145bd5a9
Weijia Shi
admin_assigned
2024-09-04T08:05:44.148Z
6400f2ed568dbe30c9161e47
/avatars/c55938df5bce82b5d96e592a1ec36a8b.svg
Weijia Shi
user
swj0419
66d7c90fe1aee6b7145bd5aa
Pete Walsh
66d7c90fe1aee6b7145bd5ab
Oyvind Tafjord
admin_assigned
2024-09-04T08:05:54.636Z
609b1aeff7c1303cfd1760f3
/avatars/3cb448514885656466886d7c84e12af1.svg
Oyvind Tafjord
user
OyvindTafjord
66d7c90fe1aee6b7145bd5ac
Nathan Lambert
admin_assigned
2024-09-04T08:06:03.121Z
628e5f90a9a3c754c1f7c88f
https://cdn-avatars.huggingface.co/v1/production/uploads/628e5f90a9a3c754c1f7c88f/iWqMY_l6dalrgRaJZWbK3.png
Nathan Lambert
user
natolambert
66d7c90fe1aee6b7145bd5ad
Yuling Gu
admin_assigned
2024-09-04T08:06:11.050Z
62a8b91c1277a49db0c5c080
/avatars/9cf834a02b67af862659c9b9d4367a52.svg
Yuling Gu
user
SaveBertAndGpt
66d7c90fe1aee6b7145bd5ae
Shane Arora
admin_assigned
2024-09-04T08:06:26.945Z
65c27f58fe2f2618f82a69bf
/avatars/53c79c0d09f36e33b8557f649d2b7261.svg
Shane Arora
user
shanearora
66d7c90fe1aee6b7145bd5af
Akshita Bhagia
admin_assigned
2024-09-04T08:06:32.853Z
6282937a9624ab5f75c977e1
https://cdn-avatars.huggingface.co/v1/production/uploads/6282937a9624ab5f75c977e1/ZOysQbGzEzmrN_SaTQgOP.jpeg
Akshita Bhagia
user
akshitab
66d7c90fe1aee6b7145bd5b0
Dustin Schwenk
admin_assigned
2024-09-04T08:06:38.370Z
647e791b7f9ad5e44babb965
https://cdn-avatars.huggingface.co/v1/production/uploads/647e791b7f9ad5e44babb965/Vjfy4c3ooDwU24N7mvRdy.jpeg
Dustin Schwenk
user
drschwenk
66d7c90fe1aee6b7145bd5b1
David Wadden
admin_assigned
2024-09-04T08:06:43.850Z
5ff4e2a1463be69ae4bd42bd
https://cdn-avatars.huggingface.co/v1/production/uploads/1609884433338-5ff4e2a1463be69ae4bd42bd.jpeg
David Wadden
user
dwadden
66d7c90fe1aee6b7145bd5b2
Alexander Wettig
admin_assigned
2024-09-04T08:06:50.080Z
6491fd16fe1c0c0843df62c0
/avatars/a049fe3ab75cb91fa95c33064cf1a4d9.svg
Alexander Wettig
user
awettig
66d7c90fe1aee6b7145bd5b3
Binyuan Hui
admin_assigned
2024-09-04T08:06:56.130Z
61e4c4ca1ab24785ac11ba69
https://cdn-avatars.huggingface.co/v1/production/uploads/61e4c4ca1ab24785ac11ba69/1Q1zhhyGSJ9RJG9MzwxVv.jpeg
Binyuan Hui
user
huybery
66d7c90fe1aee6b7145bd5b4
Tim Dettmers
admin_assigned
2024-09-04T08:07:03.191Z
61a8f52d89ea472e600a0a10
https://cdn-avatars.huggingface.co/v1/production/uploads/1638462736111-noauth.png
Tim Dettmers
user
timdettmers
66d7c90fe1aee6b7145bd5b5
Douwe Kiela
admin_assigned
2024-09-04T08:07:09.260Z
61dc997715b47073db1620dc
https://cdn-avatars.huggingface.co/v1/production/uploads/1641847245435-61dc997715b47073db1620dc.jpeg
Douwe Kiela
user
douwekiela
66d7c90fe1aee6b7145bd5b6
Ali Farhadi
admin_assigned
2024-09-04T08:07:15.914Z
6660d4c1818c5c5ca0f31266
/avatars/1d2972894cb3b9df1900fdb162d9c364.svg
alifarhadi 
user
alifarhadi051
66d7c90fe1aee6b7145bd5b7
Noah A. Smith
66d7c90fe1aee6b7145bd5b8
Pang Wei Koh
admin_assigned
2024-09-04T08:07:28.876Z
641b4263abfce26bcf7b27de
/avatars/e91b4205e4f74b0dd8c333c23203a924.svg
Pang Wei Koh
user
pangwei
66d7c90fe1aee6b7145bd5b9
Amanpreet Singh
66d7c90fe1aee6b7145bd5ba
Hannaneh Hajishirzi
extracted_pending
2024-09-04T02:42:24.466Z
65ea4989fddceb72b1082fc0
/avatars/0b4186eb8bf3fa00989e3f5015550d54.svg
Hanna Hajishirzi
user
hanna-ai2
OLMoE: Open Mixture-of-Experts Language Models
We introduce OLMoE, a fully open, state-of-the-art language model leveraging
sparse Mixture-of-Experts (MoE). OLMoE-1B-7B has 7 billion (B) parameters but
uses only 1B per input token. We pretrain it on 5 trillion tokens and further
adapt it to create OLMoE-1B-7B-Instruct. Our models outperform all available
models with similar active parameters, even surpassing larger ones like
Llama2-13B-Chat and DeepSeekMoE-16B. We present various experiments on MoE
training, analyze routing in our model showing high specialization, and
open-source all aspects of our work: model weights, training data, code, and
logs.
A sparse Mixture-of-Experts language model with 7 billion parameters achieves superior performance by using only 1 billion parameters per input token and outperforms larger models in various experiments.
sparse Mixture-of-Experts OLMoE-1B-7B pretraining OLMoE-1B-7B-Instruct Llama2-13B-Chat DeepSeekMoE-16B