CroissantLLM: A Truly Bilingual French-English Language Model
We introduce CroissantLLM, a 1.3B language model pretrained on a set of 3T
English and French tokens, to bring to the research and industrial community a
high-performance, fully open-sourced bilingual model that runs swiftly on
consumer-grade local hardware. To that end, we pioneer the approach of training
an intrinsically bilingual model with a 1:1 English-to-French pretraining data
ratio, a custom tokenizer, and bilingual finetuning datasets. We release the
training dataset, notably containing a French split with manually curated,
high-quality, and varied data sources. To assess performance outside of
English, we craft a novel benchmark, FrenchBench, consisting of an array of
classification and generation tasks, covering various orthogonal aspects of
model performance in the French Language. Additionally, rooted in transparency
and to foster further Large Language Model research, we release codebases, and
dozens of checkpoints across various model sizes, training data distributions,
and training steps, as well as fine-tuned Chat models, and strong translation
models. We evaluate our model through the FMTI framework, and validate 81 % of
the transparency criteria, far beyond the scores of even most open initiatives.
This work enriches the NLP landscape, breaking away from previous
English-centric work in order to strengthen our understanding of
multilinguality in language models.
65bc67dce168a9cc4bc436b3
Manuel Faysse
extracted_confirmed
2024-02-27T10:03:04.135Z
60f2e021adf471cbdf8bb660
https://cdn-avatars.huggingface.co/v1/production/uploads/1654090481550-60f2e021adf471cbdf8bb660.jpeg
Manuel Faysse
user
manu
65bc67dce168a9cc4bc436b4
Patrick Fernandes
admin_assigned
2024-02-02T11:31:07.251Z
6354206b12edd0ed5dc235f4
/avatars/c2f5872eb4cad979635293c0af47db03.svg
Patrick Fernandes
user
patricksf
65bc67dce168a9cc4bc436b5
Nuno Guerreiro
admin_assigned
2024-02-02T11:31:25.381Z
61f06c8ba57920a251ec1a5c
https://cdn-avatars.huggingface.co/v1/production/uploads/61f06c8ba57920a251ec1a5c/TmmSN9xWm1noLeAtsUSRk.jpeg
Nuno M. Guerreiro
user
nunonmg
65bc67dce168a9cc4bc436b6
António Loison
admin_assigned
2024-02-02T11:31:33.241Z
61b3caeee49318df54457dc8
/avatars/2f22de54b67c7ec76a5e3f07318b0916.svg
António Loison
user
antoloison
65bc67dce168a9cc4bc436b7
Duarte Alves
admin_assigned
2024-02-02T11:31:40.248Z
64132452d8a418df415a6ded
https://cdn-avatars.huggingface.co/v1/production/uploads/64132452d8a418df415a6ded/qkjL5G89uldHUXlCI3n4f.jpeg
Duarte Alves
user
DuarteMRAlves
65bc67dce168a9cc4bc436b8
Caio Corro
65bc67dce168a9cc4bc436b9
Nicolas Boizard
claimed_verified
2024-02-02T09:00:42.359Z
62be186a5f59ff2320e6e32b
https://cdn-avatars.huggingface.co/v1/production/uploads/62be186a5f59ff2320e6e32b/W_emoC2uItM-MJZyCfIKI.png
Nicolas-BZRD
user
Nicolas-BZRD
65bc67dce168a9cc4bc436ba
João Alves
claimed_verified
2024-02-02T11:18:37.546Z
61ae32477b23f65fc71a62e4
/avatars/10054d6a17e65e5509f4f48cf4659cde.svg
João Alves
user
jalves97
65bc67dce168a9cc4bc436bb
Ricardo Rei
claimed_verified
2024-02-02T11:10:31.846Z
60d9a9791fa5d458da77754d
https://cdn-avatars.huggingface.co/v1/production/uploads/1624887555363-60d9a9791fa5d458da77754d.jpeg
Ricardo Rei
user
RicardoRei
65bc67dce168a9cc4bc436bc
Pedro Martins
65bc67dce168a9cc4bc436bd
Antoni Bigata Casademunt
claimed_verified
2024-02-02T11:25:23.341Z
640777812e309e65452491dd
/avatars/01b927675e78f212408168522f65fe36.svg
Antoni Bigata
user
toninio19
65bc67dce168a9cc4bc436be
François Yvon
admin_assigned
2024-02-02T11:32:17.706Z
62ab10f04bd2ebf5dbad205c
/avatars/65356b3b057159cc67a86efb26b53486.svg
François Yvon
user
fyvo
65bc67dce168a9cc4bc436bf
André Martins
65bc67dce168a9cc4bc436c0
Gautier Viaud
admin_assigned
2024-02-02T11:33:15.397Z
618a65e17304dc918c6602ff
/avatars/8af1112094169da80c65e24ab71c7e59.svg
Gautier Viaud
user
gautierviaud
65bc67dce168a9cc4bc436c1
Céline Hudelot
65bc67dce168a9cc4bc436c2
Pierre Colombo
admin_assigned
2024-02-02T11:33:37.095Z
644a900e3a619fe72b14af0f
/avatars/e2d5dac3d92757ed48e37e126a3464a3.svg
Colombo
user
PierreColombo
CroissantLLM: A Truly Bilingual French-English Language Model
We introduce CroissantLLM, a 1.3B language model pretrained on a set of 3T
English and French tokens, to bring to the research and industrial community a
high-performance, fully open-sourced bilingual model that runs swiftly on
consumer-grade local hardware. To that end, we pioneer the approach of training
an intrinsically bilingual model with a 1:1 English-to-French pretraining data
ratio, a custom tokenizer, and bilingual finetuning datasets. We release the
training dataset, notably containing a French split with manually curated,
high-quality, and varied data sources. To assess performance outside of
English, we craft a novel benchmark, FrenchBench, consisting of an array of
classification and generation tasks, covering various orthogonal aspects of
model performance in the French Language. Additionally, rooted in transparency
and to foster further Large Language Model research, we release codebases, and
dozens of checkpoints across various model sizes, training data distributions,
and training steps, as well as fine-tuned Chat models, and strong translation
models. We evaluate our model through the FMTI framework, and validate 81 % of
the transparency criteria, far beyond the scores of even most open initiatives.
This work enriches the NLP landscape, breaking away from previous
English-centric work in order to strengthen our understanding of
multilinguality in language models.
CroissantLLM, a 1.3B bilingual language model trained on balanced English and French datasets, achieves high performance on consumer hardware and is validated using custom transparency criteria in FrenchBench.
MLM CroissantLLM pretraining bilingual model custom tokenizer bilingual finetuning FrenchBench model performance FMTI framework transparency criteria multilinguality