H2O-Danube-1.8B Technical Report
We present H2O-Danube-1.8B, a 1.8B language model trained on 1T tokens
following the core principles of LLama 2 and Mistral. We leverage and refine
various techniques for pre-training large language models. Although our model
is trained on significantly fewer total tokens compared to reference models of
similar size, it exhibits highly competitive metrics across a multitude of
benchmarks. We additionally release a chat model trained with supervised
fine-tuning followed by direct preference optimization. We make H2O-Danube-1.8B
openly available under Apache 2.0 license further democratizing LLMs to a wider
audience economically.
65b9b4cf8187af6cd33b695c
Philipp Singer
admin_assigned
2024-01-31T10:13:20.392Z
636d18755aaed143cd6698ef
https://cdn-avatars.huggingface.co/v1/production/uploads/636d18755aaed143cd6698ef/AalDh13Gp8jv1BfM5IASh.png
Philipp Singer
user
psinger
65b9b4cf8187af6cd33b695d
Pascal Pfeiffer
admin_assigned
2024-01-31T10:13:40.358Z
6316fc44c92fd6fee3161e9a
https://cdn-avatars.huggingface.co/v1/production/uploads/1676417502037-6316fc44c92fd6fee3161e9a.png
Pascal Pfeiffer
user
ilu000
65b9b4cf8187af6cd33b695e
Yauhen Babakhin
admin_assigned
2024-01-31T10:13:46.762Z
643f884da649e1a499446c11
https://cdn-avatars.huggingface.co/v1/production/uploads/643f884da649e1a499446c11/4lmRszXeD1S62sfwUHg2D.jpeg
Yauhen Babakhin
user
ybabakhin
65b9b4cf8187af6cd33b695f
Maximilian Jeblick
admin_assigned
2024-01-31T10:13:53.749Z
614c9145b44c102499783617
https://cdn-avatars.huggingface.co/v1/production/uploads/1674725099569-614c9145b44c102499783617.jpeg
Maximilian Jeblick
user
MaxJeblick
65b9b4cf8187af6cd33b6960
Nischay Dhankhar
admin_assigned
2024-01-31T10:14:06.030Z
659bd735d7207fb4a2d021d9
/avatars/f9f0283231955b5034fe36861c999601.svg
Nischay Dhankhar
user
Nischay27
65b9b4cf8187af6cd33b6961
Gabor Fodor
admin_assigned
2024-01-31T10:14:31.967Z
6576c9e4e58db15694dfe69a
https://cdn-avatars.huggingface.co/v1/production/uploads/6576c9e4e58db15694dfe69a/7WYZdlpqi0TEVN24za-op.jpeg
Gabor Fodor
user
gaborfodorh2o
65b9b4cf8187af6cd33b6962
Sri Satish Ambati
admin_assigned
2024-01-31T10:14:43.015Z
60cc616faee6f81536eee756
/avatars/5022e020a9804157a6f82804b4942d51.svg
sri ambati
user
srisatish
H2O-Danube-1.8B Technical Report
We present H2O-Danube-1.8B, a 1.8B language model trained on 1T tokens
following the core principles of LLama 2 and Mistral. We leverage and refine
various techniques for pre-training large language models. Although our model
is trained on significantly fewer total tokens compared to reference models of
similar size, it exhibits highly competitive metrics across a multitude of
benchmarks. We additionally release a chat model trained with supervised
fine-tuning followed by direct preference optimization. We make H2O-Danube-1.8B
openly available under Apache 2.0 license further democratizing LLMs to a wider
audience economically.
A 1.8B parameter language model trained on 1T tokens demonstrates competitive performance across benchmarks and is released publicly with supervised fine-tuning and preference optimization.
language model pre-training supervised fine-tuning direct preference optimization LLMs