FinGPT: Large Generative Models for a Small Language
Large language models (LLMs) excel in many tasks in NLP and beyond, but most
open models have very limited coverage of smaller languages and LLM work tends
to focus on languages where nearly unlimited data is available for pretraining.
In this work, we study the challenges of creating LLMs for Finnish, a language
spoken by less than 0.1% of the world population. We compile an extensive
dataset of Finnish combining web crawls, news, social media and eBooks. We
pursue two approaches to pretrain models: 1) we train seven monolingual models
from scratch (186M to 13B parameters) dubbed FinGPT, 2) we continue the
pretraining of the multilingual BLOOM model on a mix of its original training
data and Finnish, resulting in a 176 billion parameter model we call BLUUMI.
For model evaluation, we introduce FIN-bench, a version of BIG-bench with
Finnish tasks. We also assess other model qualities such as toxicity and bias.
Our models and tools are openly available at https://turkunlp.org/gpt3-finnish.
65519918a43b13ee3479719e
Risto Luukkonen
admin_assigned
2023-11-13T10:51:53.285Z
60fab40cd0b1858cd139b745
/avatars/678b84ae7dff1115cff40efbe366ca7d.svg
Risto Luukkonen
user
rluukkon
65519918a43b13ee3479719f
Ville Komulainen
admin_assigned
2023-11-13T10:52:01.554Z
63f4d05f121f98947078baa6
/avatars/02a16a535f7aa90b6b312388293b2bbd.svg
Ville Komulainen
user
Villekom
65519918a43b13ee347971a0
Jouni Luoma
admin_assigned
2023-11-13T10:52:08.385Z
63067e7414e7d8892aadb008
/avatars/d5c2ab86bd5df07dabc055b77cf09927.svg
Jouni Luoma
user
Joffe
65519918a43b13ee347971a1
Anni Eskelinen
admin_assigned
2023-11-13T10:52:15.469Z
63ecae3454bba6781516a98b
/avatars/9d270daa837a8a9b01b0d312eb0e184d.svg
Anni Eskelinen
user
annieske
65519918a43b13ee347971a2
Jenna Kanerva
admin_assigned
2023-11-13T10:52:21.955Z
618e380849baf26530e19574
https://cdn-avatars.huggingface.co/v1/production/uploads/1636710617882-618e380849baf26530e19574.png
Jenna Kanerva
user
jmnybl
65519918a43b13ee347971a3
Hanna-Mari Kupari
65519918a43b13ee347971a4
Filip Ginter
admin_assigned
2023-11-13T10:52:39.230Z
618a6b0bd128cf918549eff8
https://cdn-avatars.huggingface.co/v1/production/uploads/1676473724392-618a6b0bd128cf918549eff8.jpeg
Filip Ginter
user
fginter
65519918a43b13ee347971a5
Veronika Laippala
admin_assigned
2023-11-13T10:52:46.290Z
6180f823067f702bd7415c03
/avatars/726112779125678e0ca8e77423bc9eb4.svg
Veronika Laippala
user
VeronikaLaippala
65519918a43b13ee347971a6
Niklas Muennighoff
admin_assigned
2023-11-13T10:52:53.430Z
5f1eb362eec0ad2a071ad6e2
https://cdn-avatars.huggingface.co/v1/production/uploads/5f1eb362eec0ad2a071ad6e2/IXMYkYKuTwn6kBdWnQeeY.png
Niklas Muennighoff
user
Muennighoff
65519918a43b13ee347971a7
Aleksandra Piktus
admin_assigned
2023-11-13T10:53:00.628Z
5f5f4f093c67af20d9945a6c
https://cdn-avatars.huggingface.co/v1/production/uploads/1600081648206-noauth.jpeg
Aleksandra Piktus
user
ola13
65519918a43b13ee347971a8
Thomas Wang
admin_assigned
2023-11-16T16:56:20.423Z
61140e35d08630d2676c9829
/avatars/3d6e4b4d02eda7b5ef28e1cc0fb8e08a.svg
Thomas Wang
user
TimeRobber
65519918a43b13ee347971a9
Nouamane Tazi
admin_assigned
2023-11-13T10:53:34.607Z
5ff8c9f4b2035d9a81a859f7
https://cdn-avatars.huggingface.co/v1/production/uploads/1652134289581-5ff8c9f4b2035d9a81a859f7.jpeg
Nouamane Tazi
user
nouamanetazi
65519918a43b13ee347971aa
Teven Le Scao
admin_assigned
2023-11-13T10:53:45.537Z
5e67bed6100906368940747b
https://cdn-avatars.huggingface.co/v1/production/uploads/1583858339980-5e67bed6100906368940747b.jpeg
Teven Le Scao
user
teven
65519918a43b13ee347971ab
Thomas Wolf
admin_assigned
2023-11-13T10:54:13.866Z
5df7e9e5da6d0311fd3d53f9
https://cdn-avatars.huggingface.co/v1/production/uploads/1583857746553-5df7e9e5da6d0311fd3d53f9.jpeg
Thomas Wolf
user
thomwolf
65519918a43b13ee347971ac
Osma Suominen
admin_assigned
2023-11-13T10:55:06.920Z
633dc4e6c6e26ad2575e1799
https://cdn-avatars.huggingface.co/v1/production/uploads/633dc4e6c6e26ad2575e1799/LuRAZ_B_tLYJl0uzy0OkY.jpeg
Osma Suominen
user
osma
65519918a43b13ee347971ad
Samuli Sairanen
65519918a43b13ee347971ae
Mikko Merioksa
65519918a43b13ee347971af
Jyrki Heinonen
65519918a43b13ee347971b0
Aija Vahtola
65519918a43b13ee347971b1
Samuel Antao
65519918a43b13ee347971b2
Sampo Pyysalo
extracted_confirmed
2024-04-02T08:08:25.122Z
602fb2d93d343696d96ef287
/avatars/a78ce59506a416e3df1d283a13e5aefa.svg
Sampo Pyysalo
user
spyysalo
FinGPT: Large Generative Models for a Small Language
Large language models (LLMs) excel in many tasks in NLP and beyond, but most
open models have very limited coverage of smaller languages and LLM work tends
to focus on languages where nearly unlimited data is available for pretraining.
In this work, we study the challenges of creating LLMs for Finnish, a language
spoken by less than 0.1% of the world population. We compile an extensive
dataset of Finnish combining web crawls, news, social media and eBooks. We
pursue two approaches to pretrain models: 1) we train seven monolingual models
from scratch (186M to 13B parameters) dubbed FinGPT, 2) we continue the
pretraining of the multilingual BLOOM model on a mix of its original training
data and Finnish, resulting in a 176 billion parameter model we call BLUUMI.
For model evaluation, we introduce FIN-bench, a version of BIG-bench with
Finnish tasks. We also assess other model qualities such as toxicity and bias.
Our models and tools are openly available at https://turkunlp.org/gpt3-finnish.
The study addresses the challenges of creating large language models for underrepresented languages like Finnish, developing both monolingual and multilingual models, and evaluating their performance through a newly created benchmark.
large language models LLMs NLP monolingual models multilingual models pretraining dataset FinGPT BLOOM BLUUMI BIG-bench FIN-bench toxicity bias