Bielik v3 Small: Technical Report
We introduce Bielik v3, a series of parameter-efficient generative text
models (1.5B and 4.5B) optimized for Polish language processing. These models
demonstrate that smaller, well-optimized architectures can achieve performance
comparable to much larger counterparts while requiring substantially fewer
computational resources. Our approach incorporates several key innovations: a
custom Polish tokenizer (APT4) that significantly improves token efficiency,
Weighted Instruction Cross-Entropy Loss to balance learning across instruction
types, and Adaptive Learning Rate that dynamically adjusts based on training
progress. Trained on a meticulously curated corpus of 292 billion tokens
spanning 303 million documents, these models excel across multiple benchmarks,
including the Open PL LLM Leaderboard, Complex Polish Text Understanding
Benchmark, Polish EQ-Bench, and Polish Medical Leaderboard. The 4.5B parameter
model achieves results competitive with models 2-3 times its size, while the
1.5B model delivers strong performance despite its extremely compact profile.
These advances establish new benchmarks for parameter-efficient language
modeling in less-represented languages, making high-quality Polish language AI
more accessible for resource-constrained applications.
6819ef0b2ff435c58da4d860
Krzysztof Ociepa
claimed_verified
2025-05-12T09:03:56.213Z
63ecbccac8827dd0f0f59579
https://cdn-avatars.huggingface.co/v1/production/uploads/63ecbccac8827dd0f0f59579/kz-2F9Z0QKllifgZmr8tH.jpeg
Chris Ociepa
user
chrisociepa
6819ef0b2ff435c58da4d861
Łukasz Flis
claimed_verified
2025-05-14T07:36:18.501Z
6609d7a8362a1d713aa4b260
/avatars/5ed9fbc9352e8664787aa56b89b96611.svg
Lukasz Flis
user
lflis
6819ef0b2ff435c58da4d862
Remigiusz Kinas
claimed_verified
2025-05-12T06:51:15.217Z
61786d0b038518aa2827c6b7
https://cdn-avatars.huggingface.co/v1/production/uploads/61786d0b038518aa2827c6b7/d1UnfivoVreYebS5JM3P9.jpeg
Remek Kinas
user
Remek
6819ef0b2ff435c58da4d863
Krzysztof Wróbel
claimed_verified
2025-05-12T09:03:54.135Z
5e47d3eb178ca95365287400
https://cdn-avatars.huggingface.co/v1/production/uploads/1633554153914-5e47d3eb178ca95365287400.png
Krzysztof Wróbel
user
djstrong
6819ef0b2ff435c58da4d864
Adrian Gwoździej
claimed_verified
2025-05-12T12:56:24.966Z
636b9b006890dd5a450081a5
/avatars/2b4ba89894d952f95ecd1e9926580608.svg
AG
user
adgw
Bielik v3 Small: Technical Report
We introduce Bielik v3, a series of parameter-efficient generative text
models (1.5B and 4.5B) optimized for Polish language processing. These models
demonstrate that smaller, well-optimized architectures can achieve performance
comparable to much larger counterparts while requiring substantially fewer
computational resources. Our approach incorporates several key innovations: a
custom Polish tokenizer (APT4) that significantly improves token efficiency,
Weighted Instruction Cross-Entropy Loss to balance learning across instruction
types, and Adaptive Learning Rate that dynamically adjusts based on training
progress. Trained on a meticulously curated corpus of 292 billion tokens
spanning 303 million documents, these models excel across multiple benchmarks,
including the Open PL LLM Leaderboard, Complex Polish Text Understanding
Benchmark, Polish EQ-Bench, and Polish Medical Leaderboard. The 4.5B parameter
model achieves results competitive with models 2-3 times its size, while the
1.5B model delivers strong performance despite its extremely compact profile.
These advances establish new benchmarks for parameter-efficient language
modeling in less-represented languages, making high-quality Polish language AI
more accessible for resource-constrained applications.
Bielik v3, a series of parameter-efficient generative text models, achieves high performance in Polish language processing with a custom tokenizer, Weighted Instruction Cross-Entropy Loss, and Adaptive Learning Rate.
parameter-efficient generative text models custom Polish tokenizer Weighted Instruction Cross-Entropy Loss Adaptive Learning Rate Open PL LLM Leaderboard Complex Polish Text Understanding Benchmark Polish EQ-Bench Polish Medical Leaderboard