JetMoE: Reaching Llama2 Performance with 0.1M Dollars
Large Language Models (LLMs) have achieved remarkable results, but their
increasing resource demand has become a major obstacle to the development of
powerful and accessible super-human intelligence. This report introduces
JetMoE-8B, a new LLM trained with less than $0.1 million, using 1.25T tokens
from carefully mixed open-source corpora and 30,000 H100 GPU hours. Despite its
low cost, the JetMoE-8B demonstrates impressive performance, with JetMoE-8B
outperforming the Llama2-7B model and JetMoE-8B-Chat surpassing the
Llama2-13B-Chat model. These results suggest that LLM training can be much more
cost-effective than generally thought. JetMoE-8B is based on an efficient
Sparsely-gated Mixture-of-Experts (SMoE) architecture, composed of attention
and feedforward experts. Both layers are sparsely activated, allowing JetMoE-8B
to have 8B parameters while only activating 2B for each input token, reducing
inference computation by about 70% compared to Llama2-7B. Moreover, JetMoE-8B
is highly open and academia-friendly, using only public datasets and training
code. All training parameters and data mixtures have been detailed in this
report to facilitate future efforts in the development of open foundation
models. This transparency aims to encourage collaboration and further
advancements in the field of accessible and efficient LLMs. The model weights
are publicly available at https://github.com/myshell-ai/JetMoE.
66188b1112e9f8ef47c1bacf
Yikang Shen
extracted_confirmed
2024-06-25T18:40:33.328Z
6463995939359568c63d740c
/avatars/11caacefc68c1259dbc8764cc4340481.svg
Yikang Shen
user
YikangS
66188b1112e9f8ef47c1bad0
Zhen Guo
claimed_verified
2024-04-12T03:05:05.402Z
638e4e66629b4d0a62ce1bf3
https://cdn-avatars.huggingface.co/v1/production/uploads/638e4e66629b4d0a62ce1bf3/s7uQ2qmee2CaXfjZDOovZ.jpeg
Zhen Guo
user
zguo0525
66188b1112e9f8ef47c1bad1
Tianle Cai
admin_assigned
2024-04-18T12:09:49.339Z
64459357c525660aa20be337
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/1fryBXRuSvAgMIDkpWBpI.jpeg
Tianle Cai
user
tianlecai
66188b1112e9f8ef47c1bad2
Zengyi Qin
admin_assigned
2024-04-18T12:09:55.455Z
641de0213239b631552713e4
https://cdn-avatars.huggingface.co/v1/production/uploads/641de0213239b631552713e4/nRul74VIwPqVOhvzcBfVH.jpeg
Zengyi Qin
user
Zengyi-Qin
JetMoE: Reaching Llama2 Performance with 0.1M Dollars
Large Language Models (LLMs) have achieved remarkable results, but their
increasing resource demand has become a major obstacle to the development of
powerful and accessible super-human intelligence. This report introduces
JetMoE-8B, a new LLM trained with less than $0.1 million, using 1.25T tokens
from carefully mixed open-source corpora and 30,000 H100 GPU hours. Despite its
low cost, the JetMoE-8B demonstrates impressive performance, with JetMoE-8B
outperforming the Llama2-7B model and JetMoE-8B-Chat surpassing the
Llama2-13B-Chat model. These results suggest that LLM training can be much more
cost-effective than generally thought. JetMoE-8B is based on an efficient
Sparsely-gated Mixture-of-Experts (SMoE) architecture, composed of attention
and feedforward experts. Both layers are sparsely activated, allowing JetMoE-8B
to have 8B parameters while only activating 2B for each input token, reducing
inference computation by about 70% compared to Llama2-7B. Moreover, JetMoE-8B
is highly open and academia-friendly, using only public datasets and training
code. All training parameters and data mixtures have been detailed in this
report to facilitate future efforts in the development of open foundation
models. This transparency aims to encourage collaboration and further
advancements in the field of accessible and efficient LLMs. The model weights
are publicly available at https://github.com/myshell-ai/JetMoE.
JetMoE-8B, a cost-effective large language model with 8 billion parameters, achieves impressive performance using a Sparsely-gated Mixture-of-Experts architecture, demonstrating efficient use of resources and computational savings.
Sparsely-gated Mixture-of-Experts SMoE attention experts feedforward experts