UnifiedCrawl: Aggregated Common Crawl for Affordable Adaptation of LLMs
  on Low-Resource Languages
Large language models (LLMs) under-perform on low-resource languages due to
limited training data. We present a method to efficiently collect text data for
low-resource languages from the entire Common Crawl corpus. Our approach,
UnifiedCrawl, filters and extracts common crawl using minimal compute
resources, yielding mono-lingual datasets much larger than previously available
sources. We demonstrate that leveraging this data to fine-tuning multilingual
LLMs via efficient adapter methods (QLoRA) significantly boosts performance on
the low-resource language, while minimizing VRAM usage. Our experiments show
large improvements in language modeling perplexity and an increase in few-shot
prompting scores. Our work and released source code provide an affordable
approach to improve LLMs for low-resource languages using consumer hardware.
Our source code is available here at
https://github.com/bethelmelesse/unifiedcrawl.
674061fa1b8664981af12c10
Bethel Melesse Tessema
admin_assigned
2024-11-22T13:19:03.955Z
63668e73cbf2cf329187b45d
/avatars/814ebbcc9d251c6bc526309714a9baef.svg
Bethel Melesse Tessema
user
bethelmelesse
674061fa1b8664981af12c11
Akhil Kedia
674061fa1b8664981af12c12
Tae-Sun Chung
UnifiedCrawl: Aggregated Common Crawl for Affordable Adaptation of LLMs
  on Low-Resource Languages
Large language models (LLMs) under-perform on low-resource languages due to
limited training data. We present a method to efficiently collect text data for
low-resource languages from the entire Common Crawl corpus. Our approach,
UnifiedCrawl, filters and extracts common crawl using minimal compute
resources, yielding mono-lingual datasets much larger than previously available
sources. We demonstrate that leveraging this data to fine-tuning multilingual
LLMs via efficient adapter methods (QLoRA) significantly boosts performance on
the low-resource language, while minimizing VRAM usage. Our experiments show
large improvements in language modeling perplexity and an increase in few-shot
prompting scores. Our work and released source code provide an affordable
approach to improve LLMs for low-resource languages using consumer hardware.
Our source code is available here at
https://github.com/bethelmelesse/unifiedcrawl.
UnifiedCrawl efficiently collects data from Common Crawl to fine-tune multilingual LLMs using adapter methods, enhancing performance on low-resource languages.
Large language models Common Crawl UnifiedCrawl mono-lingual datasets multilingual LLMs efficient adapter methods QLoRA VRAM usage language modeling perplexity few-shot prompting scores