Bielik 7B v0.1: A Polish Language Model -- Development, Insights, and
  Evaluation
We introduce Bielik 7B v0.1, a 7-billion-parameter generative text model for
Polish language processing. Trained on curated Polish corpora, this model
addresses key challenges in language model development through innovative
techniques. These include Weighted Instruction Cross-Entropy Loss, which
balances the learning of different instruction types, and Adaptive Learning
Rate, which dynamically adjusts the learning rate based on training progress.
To evaluate performance, we created the Open PL LLM Leaderboard and Polish
MT-Bench, novel frameworks assessing various NLP tasks and conversational
abilities. Bielik 7B v0.1 demonstrates significant improvements, achieving a 9
percentage point increase in average score compared to Mistral-7B-v0.1 on the
RAG Reader task. It also excels in the Polish MT-Bench, particularly in
Reasoning (6.15/10) and Role-playing (7.83/10) categories. This model
represents a substantial advancement in Polish language AI, offering a powerful
tool for diverse linguistic applications and setting new benchmarks in the
field.
671cd85ad5e54b67236902c8
Krzysztof Ociepa
claimed_verified
2024-10-28T11:24:02.294Z
63ecbccac8827dd0f0f59579
https://cdn-avatars.huggingface.co/v1/production/uploads/63ecbccac8827dd0f0f59579/kz-2F9Z0QKllifgZmr8tH.jpeg
Chris Ociepa
user
chrisociepa
671cd85ad5e54b67236902c9
Łukasz Flis
claimed_verified
2024-10-28T15:14:45.213Z
6609d7a8362a1d713aa4b260
/avatars/5ed9fbc9352e8664787aa56b89b96611.svg
Lukasz Flis
user
lflis
671cd85ad5e54b67236902ca
Krzysztof Wróbel
claimed_verified
2024-10-28T10:36:20.109Z
5e47d3eb178ca95365287400
https://cdn-avatars.huggingface.co/v1/production/uploads/1633554153914-5e47d3eb178ca95365287400.png
Krzysztof Wróbel
user
djstrong
671cd85ad5e54b67236902cb
Adrian Gwoździej
claimed_verified
2024-10-28T11:24:14.679Z
636b9b006890dd5a450081a5
/avatars/2b4ba89894d952f95ecd1e9926580608.svg
AG
user
adgw
671cd85ad5e54b67236902cc
Remigiusz Kinas
claimed_verified
2024-10-28T11:24:08.369Z
61786d0b038518aa2827c6b7
https://cdn-avatars.huggingface.co/v1/production/uploads/61786d0b038518aa2827c6b7/d1UnfivoVreYebS5JM3P9.jpeg
Remek Kinas
user
Remek
Bielik 7B v0.1: A Polish Language Model -- Development, Insights, and
  Evaluation
We introduce Bielik 7B v0.1, a 7-billion-parameter generative text model for
Polish language processing. Trained on curated Polish corpora, this model
addresses key challenges in language model development through innovative
techniques. These include Weighted Instruction Cross-Entropy Loss, which
balances the learning of different instruction types, and Adaptive Learning
Rate, which dynamically adjusts the learning rate based on training progress.
To evaluate performance, we created the Open PL LLM Leaderboard and Polish
MT-Bench, novel frameworks assessing various NLP tasks and conversational
abilities. Bielik 7B v0.1 demonstrates significant improvements, achieving a 9
percentage point increase in average score compared to Mistral-7B-v0.1 on the
RAG Reader task. It also excels in the Polish MT-Bench, particularly in
Reasoning (6.15/10) and Role-playing (7.83/10) categories. This model
represents a substantial advancement in Polish language AI, offering a powerful
tool for diverse linguistic applications and setting new benchmarks in the
field.
Bielik 7B v0.1, a 7-billion-parameter generative text model for Polish, uses Weighted Instruction Cross-Entropy Loss and Adaptive Learning Rate to enhance performance in various NLP tasks, surpassing Mistral-7B-v0.1 and setting new benchmarks.
Weighted Instruction Cross-Entropy Loss Adaptive Learning Rate Open PL LLM Leaderboard Polish MT-Bench generative text model NLP tasks RAG Reader Reasoning Role-playing