Small Models, Big Impact: Efficient Corpus and Graph-Based Adaptation of
  Small Multilingual Language Models for Low-Resource Languages
Low-resource languages (LRLs) face significant challenges in natural language
processing (NLP) due to limited data. While current state-of-the-art large
language models (LLMs) still struggle with LRLs, smaller multilingual models
(mLMs) such as mBERT and XLM-R offer greater promise due to a better fit of
their capacity to low training data sizes. This study systematically
investigates parameter-efficient adapter-based methods for adapting mLMs to
LRLs, evaluating three architectures: Sequential Bottleneck, Invertible
Bottleneck, and Low-Rank Adaptation. Using unstructured text from GlotCC and
structured knowledge from ConceptNet, we show that small adaptation datasets
(e.g., up to 1 GB of free-text or a few MB of knowledge graph data) yield gains
in intrinsic (masked language modeling) and extrinsic tasks (topic
classification, sentiment analysis, and named entity recognition). We find that
Sequential Bottleneck adapters excel in language modeling, while Invertible
Bottleneck adapters slightly outperform other methods on downstream tasks due
to better embedding alignment and larger parameter counts. Adapter-based
methods match or outperform full fine-tuning while using far fewer parameters,
and smaller mLMs prove more effective for LRLs than massive LLMs like LLaMA-3,
GPT-4, and DeepSeek-R1-based distilled models. While adaptation improves
performance, pre-training data size remains the dominant factor, especially for
languages with extensive pre-training coverage.
67b32e9dff65b4ec02cb6d81
Daniil Gurgurov
claimed_verified
2025-02-18T16:39:55.120Z
6427f45beb320ead3d287acf
https://cdn-avatars.huggingface.co/v1/production/uploads/6427f45beb320ead3d287acf/J7DLdQM6ehiJBke112s0q.jpeg
Daniil Gurgurov
user
DGurgurov
67b32e9dff65b4ec02cb6d82
Ivan Vykopal
extracted_pending
2025-02-17T12:42:07.190Z
64b7e5a89e7deb6a7824301b
/avatars/34d8843fe6203c4d09bf1b442405a4ab.svg
Ivan Vykopal
user
ivykopal
67b32e9dff65b4ec02cb6d83
Josef van Genabith
67b32e9dff65b4ec02cb6d84
Simon Ostermann
claimed_verified
2025-02-21T10:00:05.016Z
664de652b3eaebda418c4ce9
/avatars/7f35afa838566daaeebb6e2915013871.svg
Simon Ostermann
user
simonost
Small Models, Big Impact: Efficient Corpus and Graph-Based Adaptation of
  Small Multilingual Language Models for Low-Resource Languages
Low-resource languages (LRLs) face significant challenges in natural language
processing (NLP) due to limited data. While current state-of-the-art large
language models (LLMs) still struggle with LRLs, smaller multilingual models
(mLMs) such as mBERT and XLM-R offer greater promise due to a better fit of
their capacity to low training data sizes. This study systematically
investigates parameter-efficient adapter-based methods for adapting mLMs to
LRLs, evaluating three architectures: Sequential Bottleneck, Invertible
Bottleneck, and Low-Rank Adaptation. Using unstructured text from GlotCC and
structured knowledge from ConceptNet, we show that small adaptation datasets
(e.g., up to 1 GB of free-text or a few MB of knowledge graph data) yield gains
in intrinsic (masked language modeling) and extrinsic tasks (topic
classification, sentiment analysis, and named entity recognition). We find that
Sequential Bottleneck adapters excel in language modeling, while Invertible
Bottleneck adapters slightly outperform other methods on downstream tasks due
to better embedding alignment and larger parameter counts. Adapter-based
methods match or outperform full fine-tuning while using far fewer parameters,
and smaller mLMs prove more effective for LRLs than massive LLMs like LLaMA-3,
GPT-4, and DeepSeek-R1-based distilled models. While adaptation improves
performance, pre-training data size remains the dominant factor, especially for
languages with extensive pre-training coverage.
Adapter-based methods improve multilingual models' performance on low-resource languages using minimal adaptation data, outperforming full fine-tuning with fewer parameters.
parameter-efficient adapter-based methods Sequential Bottleneck Invertible Bottleneck Low-Rank Adaptation masked language modeling topic classification sentiment analysis named entity recognition embedding alignment pre-training data size low-resource languages multilingual models mBERT XLM-R LLaMA-3 GPT-4 DeepSeek-R1-based distilled models