Pensez: Less Data, Better Reasoning -- Rethinking French LLM
Large language models (LLMs) have demonstrated remarkable capabilities in
various natural language processing tasks. However, achieving strong
performance in specialized domains like mathematical reasoning and non-English
languages often requires extensive training on massive datasets. This paper
investigates a contrasting approach: strategic fine-tuning on a small,
high-quality, bilingual (English-French) dataset to enhance both the reasoning
capabilities and French language proficiency of a large language model. Rather
than relying on scale, we explore the hypothesis that targeted data curation
and optimized training can achieve competitive, or even superior, performance.
We demonstrate, through targeted supervised fine-tuning (SFT) on only 2,000
carefully selected samples, significant improvements in mathematical reasoning.
Specifically, Pensez 7B exhibits an increase in accuracy of the base model up
to 20% on the AIME25 and a 12% increase on a French MATH level 5 benchmark.
These results challenge the prevailing assumption that massive datasets are
aprerequisite for strong reasoning performance in LLMs, highlighting the
potential of strategic data curation and optimized fine-tuning for enhancing
both specialized skills and multilingual capabilities. Our findings have
implications for the efficient development of high-performing, multilingual
LLMs, especially in resource-constrained scenarios.
67da8a9fdab8cc723c349fb0
Huy Hoang Ha
claimed_verified
2025-03-19T09:43:52.541Z
630a5ef0e81e1dea2cedcec0
https://cdn-avatars.huggingface.co/v1/production/uploads/630a5ef0e81e1dea2cedcec0/ATtyCvYoX4z7uxsm2sJU2.png
Hà Huy Hoàng
user
HoangHa
Pensez: Less Data, Better Reasoning -- Rethinking French LLM
Large language models (LLMs) have demonstrated remarkable capabilities in
various natural language processing tasks. However, achieving strong
performance in specialized domains like mathematical reasoning and non-English
languages often requires extensive training on massive datasets. This paper
investigates a contrasting approach: strategic fine-tuning on a small,
high-quality, bilingual (English-French) dataset to enhance both the reasoning
capabilities and French language proficiency of a large language model. Rather
than relying on scale, we explore the hypothesis that targeted data curation
and optimized training can achieve competitive, or even superior, performance.
We demonstrate, through targeted supervised fine-tuning (SFT) on only 2,000
carefully selected samples, significant improvements in mathematical reasoning.
Specifically, Pensez 7B exhibits an increase in accuracy of the base model up
to 20% on the AIME25 and a 12% increase on a French MATH level 5 benchmark.
These results challenge the prevailing assumption that massive datasets are
aprerequisite for strong reasoning performance in LLMs, highlighting the
potential of strategic data curation and optimized fine-tuning for enhancing
both specialized skills and multilingual capabilities. Our findings have
implications for the efficient development of high-performing, multilingual
LLMs, especially in resource-constrained scenarios.
Strategic fine-tuning on a small, high-quality bilingual dataset significantly enhances the mathematical reasoning and French language capabilities of a large language model.
large language models specialized domains mathematical reasoning non-English languages strategic fine-tuning bilingual (English-French) dataset targeted data curation optimized training supervised fine-tuning (SFT) AIME25 French MATH level 5 benchmark multilingual LLMs resource-constrained scenarios