MADLAD-400: A Multilingual And Document-Level Large Audited Dataset
We introduce MADLAD-400, a manually audited, general domain 3T token
monolingual dataset based on CommonCrawl, spanning 419 languages. We discuss
the limitations revealed by self-auditing MADLAD-400, and the role data
auditing had in the dataset creation process. We then train and release a
10.7B-parameter multilingual machine translation model on 250 billion tokens
covering over 450 languages using publicly available data, and find that it is
competitive with models that are significantly larger, and report the results
on different domains. In addition, we train a 8B-parameter language model, and
assess the results on few-shot translation. We make the baseline models
available to the research community.
64ffc9041e14749e84de6190
Sneha Kudugunta
64ffc9041e14749e84de6191
Isaac Caswell
admin_assigned
2023-09-12T11:09:19.931Z
617c7165e1dfc7bbb3375abe
/avatars/f628fb21c35eaaec5ff9519c82848c44.svg
Isaac Caswell
user
icaswell
64ffc9041e14749e84de6192
Biao Zhang
claimed_verified
2024-07-19T11:25:48.067Z
6346a600bcb5e67902ac6f44
https://cdn-avatars.huggingface.co/v1/production/uploads/6346a600bcb5e67902ac6f44/Kw-Ucmxx5S9WdlYDxQQ-z.jpeg
Biao Zhang
user
bzhanggo
64ffc9041e14749e84de6193
Xavier Garcia
64ffc9041e14749e84de6194
Christopher A. Choquette-Choo
admin_assigned
2023-09-12T12:08:59.947Z
649c5307e89a7650158f4965
/avatars/3c71b501db27c86ed686ae26ad1391cd.svg
Christopher A. Choquette-Choo
user
cchoquette
64ffc9041e14749e84de6195
Katherine Lee
admin_assigned
2023-09-12T12:09:09.147Z
612a1e924f3c2ac50a06f950
/avatars/d424605e0d48327082114b878b91b161.svg
Katherine Lee
user
katherine1ee
64ffc9041e14749e84de6196
Derrick Xin
admin_assigned
2023-09-12T12:09:17.312Z
634cea1605f736dff37abe84
/avatars/6c199de4b6e7fd1320f47eeecd1310f2.svg
Derrick Xin
user
Derr
64ffc9041e14749e84de6197
Aditya Kusupati
claimed_verified
2024-10-16T09:45:39.295Z
65dd3b03f0f26b277be1e91e
/avatars/67160c2d5241aea5c5d9b211f21f9caa.svg
Aditya Kusupati
user
adityakusupati
64ffc9041e14749e84de6198
Romi Stella
64ffc9041e14749e84de6199
Ankur Bapna
admin_assigned
2023-09-12T12:09:39.560Z
62cdba279403ec4719d384f8
/avatars/c62c7d16b5cf382cdeb7fbfb5e291917.svg
Ankur Bapna
user
ankurbpn
64ffc9041e14749e84de619a
Orhan Firat
MADLAD-400: A Multilingual And Document-Level Large Audited Dataset
We introduce MADLAD-400, a manually audited, general domain 3T token
monolingual dataset based on CommonCrawl, spanning 419 languages. We discuss
the limitations revealed by self-auditing MADLAD-400, and the role data
auditing had in the dataset creation process. We then train and release a
10.7B-parameter multilingual machine translation model on 250 billion tokens
covering over 450 languages using publicly available data, and find that it is
competitive with models that are significantly larger, and report the results
on different domains. In addition, we train a 8B-parameter language model, and
assess the results on few-shot translation. We make the baseline models
available to the research community.
A multilingual translation model trained on a large dataset of 250 billion tokens across over 450 languages outperforms larger models, with competitive results in various domains and effective few-shot translation performance.
multilingual machine translation few-shot translation CommonCrawl