SmolLM2: When Smol Goes Big -- Data-Centric Training of a Small Language
  Model
While large language models have facilitated breakthroughs in many
applications of artificial intelligence, their inherent largeness makes them
computationally expensive and challenging to deploy in resource-constrained
settings. In this paper, we document the development of SmolLM2, a
state-of-the-art "small" (1.7 billion parameter) language model (LM). To attain
strong performance, we overtrain SmolLM2 on ~11 trillion tokens of data using a
multi-stage training process that mixes web text with specialized math, code,
and instruction-following data. We additionally introduce new specialized
datasets (FineMath, Stack-Edu, and SmolTalk) at stages where we found existing
datasets to be problematically small or low-quality. To inform our design
decisions, we perform both small-scale ablations as well as a manual refinement
process that updates the dataset mixing rates at each stage based on the
performance at the previous stage. Ultimately, we demonstrate that SmolLM2
outperforms other recent small LMs including Qwen2.5-1.5B and Llama3.2-1B. To
facilitate future research on LM development as well as applications of small
LMs, we release both SmolLM2 as well as all of the datasets we prepared in the
course of this project.
67a446a9430e358f5d5ac4c3
Loubna Ben Allal
claimed_verified
2025-02-06T14:14:56.506Z
61c141342aac764ce1654e43
https://cdn-avatars.huggingface.co/v1/production/uploads/61c141342aac764ce1654e43/81AwoT5IQ_Xdw0OVw7TKu.jpeg
Loubna Ben Allal
user
loubnabnl
67a446a9430e358f5d5ac4c4
Anton Lozhkov
claimed_verified
2025-02-06T14:14:39.237Z
602e6dee60e3dd96631c906e
https://cdn-avatars.huggingface.co/v1/production/uploads/1613655355830-noauth.png
Anton Lozhkov
user
anton-l
67a446a9430e358f5d5ac4c5
Elie Bakouch
claimed_verified
2025-02-06T14:14:45.734Z
651e96991b97c9f33d26bde6
https://cdn-avatars.huggingface.co/v1/production/uploads/651e96991b97c9f33d26bde6/-Bqs6qrmz0yCfwtB2e-6q.jpeg
Elie Bakouch
user
eliebak
67a446a9430e358f5d5ac4c6
Gabriel Martín Blázquez
claimed_verified
2025-02-06T14:14:43.746Z
60f2fc91b92afccb7c34b8ed
https://cdn-avatars.huggingface.co/v1/production/uploads/60f2fc91b92afccb7c34b8ed/W2-Nay12Ef4Ltyaf8EKE9.jpeg
Gabriel Martín Blázquez
user
gabrielmbmb
67a446a9430e358f5d5ac4c7
Guilherme Penedo
claimed_verified
2025-02-06T14:14:50.085Z
62596f9e1c0a084224b93e00
https://cdn-avatars.huggingface.co/v1/production/uploads/62596f9e1c0a084224b93e00/X2aLkJ0ofhkXwAg7lXvxD.jpeg
Guilherme Penedo
user
guipenedo
67a446a9430e358f5d5ac4c8
Lewis Tunstall
admin_assigned
2025-02-06T14:16:30.456Z
5f0c746619cb630495b814fd
https://cdn-avatars.huggingface.co/v1/production/uploads/1594651707950-noauth.jpeg
Lewis Tunstall
user
lewtun
67a446a9430e358f5d5ac4c9
Andrés Marafioti
admin_assigned
2025-02-06T14:16:37.742Z
65d66b494bbd0d92b641cdbb
https://cdn-avatars.huggingface.co/v1/production/uploads/65d66b494bbd0d92b641cdbb/6-7dm7B-JxcoS1QlCPdMN.jpeg
Andres Marafioti
user
andito
67a446a9430e358f5d5ac4ca
Hynek Kydlíček
admin_assigned
2025-02-06T14:16:43.590Z
626ede24d2fa9e7d598c8709
https://cdn-avatars.huggingface.co/v1/production/uploads/626ede24d2fa9e7d598c8709/JKS8-Y2Jw87EgNQZBRswq.jpeg
Hynek Kydlicek
user
hynky
67a446a9430e358f5d5ac4cb
Agustín Piqueres Lajarín
admin_assigned
2025-02-06T14:16:49.324Z
6435d564a4bd75c62cc03701
https://cdn-avatars.huggingface.co/v1/production/uploads/6435d564a4bd75c62cc03701/7P2G_wVNB6MISp2Phh427.jpeg
Agustín Piqueres Lajarín
user
plaguss
67a446a9430e358f5d5ac4cc
Vaibhav Srivastav
claimed_verified
2025-02-06T14:14:52.239Z
61b85ce86eb1f2c5e6233736
https://cdn-avatars.huggingface.co/v1/production/uploads/1655385361868-61b85ce86eb1f2c5e6233736.jpeg
Vaibhav Srivastav
user
reach-vb
67a446a9430e358f5d5ac4cd
Joshua Lochner
claimed_verified
2025-02-06T14:14:36.878Z
61b253b7ac5ecaae3d1efe0c
https://cdn-avatars.huggingface.co/v1/production/uploads/61b253b7ac5ecaae3d1efe0c/hwiQ0uvz3t-L5a-NtBIO6.png
Joshua
user
Xenova
67a446a9430e358f5d5ac4ce
Caleb Fahlgren
admin_assigned
2025-02-06T14:16:56.849Z
648a374f00f7a3374ee64b99
https://cdn-avatars.huggingface.co/v1/production/uploads/648a374f00f7a3374ee64b99/YPwSOrronoozwHbJchPn3.jpeg
Caleb Fahlgren
user
cfahlgren1
67a446a9430e358f5d5ac4cf
Xuan-Son Nguyen
admin_assigned
2025-02-06T14:17:02.477Z
63ca214abedad7e2bf1d1517
https://cdn-avatars.huggingface.co/v1/production/uploads/1674191139776-noauth.png
Xuan-Son Nguyen
user
ngxson
67a446a9430e358f5d5ac4d0
Clémentine Fourrier
claimed_verified
2025-02-06T14:14:54.591Z
6202a599216215a22221dea9
https://cdn-avatars.huggingface.co/v1/production/uploads/1644340617257-noauth.png
Clémentine Fourrier
user
clefourrier
67a446a9430e358f5d5ac4d1
Ben Burtenshaw
claimed_verified
2025-02-06T14:14:41.918Z
62d648291fa3e4e7ae3fa6e8
https://cdn-avatars.huggingface.co/v1/production/uploads/62d648291fa3e4e7ae3fa6e8/oatOwf8Xqe5eDbCSuYqCd.png
ben burtenshaw
user
burtenshaw
67a446a9430e358f5d5ac4d2
Hugo Larcher
admin_assigned
2025-02-06T14:17:14.767Z
641cc77c92cd25302998b740
https://cdn-avatars.huggingface.co/v1/production/uploads/641cc77c92cd25302998b740/5A81W5s3ecLaLXFir52Rw.jpeg
Hugo Larcher
user
hlarcher
67a446a9430e358f5d5ac4d3
Haojun Zhao
admin_assigned
2025-02-06T14:17:33.798Z
660ed80b1889bf2cd53cab7f
/avatars/93ee6ff00668c2698ad8b6fa6f072b92.svg
Haojun Zhao
user
zzhhjjj
67a446a9430e358f5d5ac4d4
Cyril Zakka
admin_assigned
2025-02-06T14:17:43.679Z
66ba71a4447411b9c0e19d71
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/4f93ZrYdaKfK3F53IB51x.jpeg
Cyril
user
cyrilzakka
67a446a9430e358f5d5ac4d5
Mathieu Morlon
admin_assigned
2025-02-06T14:17:50.199Z
664d7d1e4f54c9372970e121
https://cdn-avatars.huggingface.co/v1/production/uploads/664d7d1e4f54c9372970e121/TEWsq1_zSBBeuHLu0HlDc.png
Mathieu Morlon
user
glutamatt
67a446a9430e358f5d5ac4d6
Colin Raffel
admin_assigned
2025-02-06T14:17:57.936Z
6079c29765b9d0165cb18392
https://cdn-avatars.huggingface.co/v1/production/uploads/1618592397610-noauth.jpeg
Colin Raffel
user
craffel
67a446a9430e358f5d5ac4d7
Leandro von Werra
admin_assigned
2025-02-06T11:03:02.572Z
5e48005437cb5b49818287a5
https://cdn-avatars.huggingface.co/v1/production/uploads/5e48005437cb5b49818287a5/4uCXGGui-9QifAT4qelxU.png
Leandro von Werra
user
lvwerra
67a446a9430e358f5d5ac4d8
Thomas Wolf
admin_assigned
2025-02-06T14:14:14.159Z
5df7e9e5da6d0311fd3d53f9
https://cdn-avatars.huggingface.co/v1/production/uploads/1583857746553-5df7e9e5da6d0311fd3d53f9.jpeg
Thomas Wolf
user
thomwolf
SmolLM2: When Smol Goes Big -- Data-Centric Training of a Small Language
  Model
While large language models have facilitated breakthroughs in many
applications of artificial intelligence, their inherent largeness makes them
computationally expensive and challenging to deploy in resource-constrained
settings. In this paper, we document the development of SmolLM2, a
state-of-the-art "small" (1.7 billion parameter) language model (LM). To attain
strong performance, we overtrain SmolLM2 on ~11 trillion tokens of data using a
multi-stage training process that mixes web text with specialized math, code,
and instruction-following data. We additionally introduce new specialized
datasets (FineMath, Stack-Edu, and SmolTalk) at stages where we found existing
datasets to be problematically small or low-quality. To inform our design
decisions, we perform both small-scale ablations as well as a manual refinement
process that updates the dataset mixing rates at each stage based on the
performance at the previous stage. Ultimately, we demonstrate that SmolLM2
outperforms other recent small LMs including Qwen2.5-1.5B and Llama3.2-1B. To
facilitate future research on LM development as well as applications of small
LMs, we release both SmolLM2 as well as all of the datasets we prepared in the
course of this project.
SmolLM2, a 1.7 billion parameter language model, achieves superior performance to other small models through extensive multi-stage training with specialized datasets.
language model overtraining parameter-efficient FineMath Stack-Edu SmolTalk dataset mixing rates Qwen2.5-1.5B Llama3.2-1B