AutoMathText: Autonomous Data Selection with Language Models for
  Mathematical Texts
To improve language models' proficiency in mathematical reasoning via
continual pretraining, we introduce a novel strategy that leverages base
language models for autonomous data selection. Departing from conventional
supervised fine-tuning or trained classifiers with human-annotated data, our
approach utilizes meta-prompted language models as zero-shot verifiers to
autonomously evaluate and select high-quality mathematical content, and we
release the curated open-source AutoMathText dataset encompassing over 200GB of
data. To demonstrate the efficacy of our method, we continuously pretrained a
7B-parameter Mistral language model on the AutoMathText dataset, achieving
substantial improvements in downstream performance on the MATH dataset with a
token amount reduced by orders of magnitude compared to previous continuous
pretraining works. Our method showcases a 2 times increase in pretraining token
efficiency compared to baselines, underscoring the potential of our approach in
enhancing models' mathematical reasoning capabilities. The AutoMathText dataset
is available at https://huggingface.co/datasets/math-ai/AutoMathText. The code
is available at https://github.com/yifanzhang-pro/AutoMathText.
65cadee7c4eddc15d036c8d5
Yifan Zhang
claimed_verified
2024-02-14T08:42:53.417Z
647bf082aba7062fe5c51ca9
https://cdn-avatars.huggingface.co/v1/production/uploads/647bf082aba7062fe5c51ca9/p4lY9IjHiWZETKmFq1mtU.jpeg
Yifan Zhang
user
yifAI
65cadee7c4eddc15d036c8d6
Yifan Luo
admin_assigned
2024-02-13T12:10:39.491Z
624dae2cb58d1313f8ba9a2d
https://cdn-avatars.huggingface.co/v1/production/uploads/624dae2cb58d1313f8ba9a2d/cFar_YWV8qtWUjOZeNKT0.png
Yifan Luo
user
Luobots
65cadee7c4eddc15d036c8d7
Yang Yuan
65cadee7c4eddc15d036c8d8
Andrew Chi-Chih Yao
AutoMathText: Autonomous Data Selection with Language Models for
  Mathematical Texts
To improve language models' proficiency in mathematical reasoning via
continual pretraining, we introduce a novel strategy that leverages base
language models for autonomous data selection. Departing from conventional
supervised fine-tuning or trained classifiers with human-annotated data, our
approach utilizes meta-prompted language models as zero-shot verifiers to
autonomously evaluate and select high-quality mathematical content, and we
release the curated open-source AutoMathText dataset encompassing over 200GB of
data. To demonstrate the efficacy of our method, we continuously pretrained a
7B-parameter Mistral language model on the AutoMathText dataset, achieving
substantial improvements in downstream performance on the MATH dataset with a
token amount reduced by orders of magnitude compared to previous continuous
pretraining works. Our method showcases a 2 times increase in pretraining token
efficiency compared to baselines, underscoring the potential of our approach in
enhancing models' mathematical reasoning capabilities. The AutoMathText dataset
is available at https://huggingface.co/datasets/math-ai/AutoMathText. The code
is available at https://github.com/yifanzhang-pro/AutoMathText.
A novel strategy using meta-prompted language models autonomously selects high-quality mathematical data for continual pretraining, significantly improving a language model's mathematical reasoning and reducing token usage.
meta-prompted language models zero-shot verifiers continual pretraining AutoMathText dataset Mistral language model token efficiency MATH dataset