H2O-Danube3 Technical Report
We present H2O-Danube3, a series of small language models consisting of
H2O-Danube3-4B, trained on 6T tokens and H2O-Danube3-500M, trained on 4T
tokens. Our models are pre-trained on high quality Web data consisting of
primarily English tokens in three stages with different data mixes before final
supervised tuning for chat version. The models exhibit highly competitive
metrics across a multitude of academic, chat, and fine-tuning benchmarks.
Thanks to its compact architecture, H2O-Danube3 can be efficiently run on a
modern smartphone, enabling local inference and rapid processing capabilities
even on mobile devices. We make all models openly available under Apache 2.0
license further democratizing LLMs to a wider audience economically.
669488b7ce544964b27167a2
Pascal Pfeiffer
669488b7ce544964b27167a3
Philipp Singer
669488b7ce544964b27167a4
Yauhen Babakhin
claimed_verified
2025-02-13T20:36:31.298Z
643f884da649e1a499446c11
https://cdn-avatars.huggingface.co/v1/production/uploads/643f884da649e1a499446c11/4lmRszXeD1S62sfwUHg2D.jpeg
Yauhen Babakhin
user
ybabakhin
669488b7ce544964b27167a5
Gabor Fodor
669488b7ce544964b27167a6
Nischay Dhankhar
669488b7ce544964b27167a7
Sri Satish Ambati
H2O-Danube3 Technical Report
We present H2O-Danube3, a series of small language models consisting of
H2O-Danube3-4B, trained on 6T tokens and H2O-Danube3-500M, trained on 4T
tokens. Our models are pre-trained on high quality Web data consisting of
primarily English tokens in three stages with different data mixes before final
supervised tuning for chat version. The models exhibit highly competitive
metrics across a multitude of academic, chat, and fine-tuning benchmarks.
Thanks to its compact architecture, H2O-Danube3 can be efficiently run on a
modern smartphone, enabling local inference and rapid processing capabilities
even on mobile devices. We make all models openly available under Apache 2.0
license further democratizing LLMs to a wider audience economically.
H2O-Danube3, a series of small language models, achieves high performance across various benchmarks and is efficient for local inference on smartphones.
language models pre-trained high quality Web data English tokens stages data mixes chat version supervised tuning fine-tuning benchmarks compact architecture local inference rapid processing smartphones Apache 2.0 license