MachineLearningLM: Continued Pretraining Language Models on Millions of
  Synthetic Tabular Prediction Tasks Scales In-Context ML
Large language models (LLMs) possess broad world knowledge and strong
general-purpose reasoning ability, yet they struggle to learn from many
in-context examples on standard machine learning (ML) tasks, that is, to
leverage many-shot demonstrations purely via in-context learning (ICL) without
gradient descent. We introduce MachineLearningLM, a portable
continued-pretraining framework that equips a general-purpose LLM with robust
in-context ML capability while preserving its general knowledge and reasoning
for broader chat workflows.
  Our pretraining procedure synthesizes ML tasks from millions of structural
causal models (SCMs), spanning shot counts up to 1,024. We begin with a
random-forest teacher, distilling tree-based decision strategies into the LLM
to strengthen robustness in numerical modeling. All tasks are serialized with a
token-efficient prompt, enabling 3x to 6x more examples per context window and
delivering up to 50x amortized throughput via batch inference.
  Despite a modest setup (Qwen-2.5-7B-Instruct with LoRA rank 8),
MachineLearningLM outperforms strong LLM baselines (e.g., GPT-5-mini) by an
average of about 15% on out-of-distribution tabular classification across
finance, physics, biology, and healthcare domains. It exhibits a striking
many-shot scaling law: accuracy increases monotonically as in-context
demonstrations grow from 8 to 1,024. Without any task-specific training, it
attains random-forest-level accuracy across hundreds of shots. General chat
capabilities, including knowledge and reasoning, are preserved: it achieves
75.4% on MMLU.
68c12b823912ed54cf5432e3
Haoyu Dong
claimed_verified
2025-09-12T16:17:25.540Z
637b08057ce76c3b834da15d
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/PFmI1HNcZlHF0RoLjP3c2.png
Haoyu Dong
user
HaoyuDong
68c12b823912ed54cf5432e4
Pengkun Zhang
68c12b823912ed54cf5432e5
Mingzhe Lu
68c12b823912ed54cf5432e6
Yanzhen Shen
68c12b823912ed54cf5432e7
Guolin Ke
MachineLearningLM: Continued Pretraining Language Models on Millions of
  Synthetic Tabular Prediction Tasks Scales In-Context ML
Large language models (LLMs) possess broad world knowledge and strong
general-purpose reasoning ability, yet they struggle to learn from many
in-context examples on standard machine learning (ML) tasks, that is, to
leverage many-shot demonstrations purely via in-context learning (ICL) without
gradient descent. We introduce MachineLearningLM, a portable
continued-pretraining framework that equips a general-purpose LLM with robust
in-context ML capability while preserving its general knowledge and reasoning
for broader chat workflows.
  Our pretraining procedure synthesizes ML tasks from millions of structural
causal models (SCMs), spanning shot counts up to 1,024. We begin with a
random-forest teacher, distilling tree-based decision strategies into the LLM
to strengthen robustness in numerical modeling. All tasks are serialized with a
token-efficient prompt, enabling 3x to 6x more examples per context window and
delivering up to 50x amortized throughput via batch inference.
  Despite a modest setup (Qwen-2.5-7B-Instruct with LoRA rank 8),
MachineLearningLM outperforms strong LLM baselines (e.g., GPT-5-mini) by an
average of about 15% on out-of-distribution tabular classification across
finance, physics, biology, and healthcare domains. It exhibits a striking
many-shot scaling law: accuracy increases monotonically as in-context
demonstrations grow from 8 to 1,024. Without any task-specific training, it
attains random-forest-level accuracy across hundreds of shots. General chat
capabilities, including knowledge and reasoning, are preserved: it achieves
75.4% on MMLU.
MachineLearningLM enhances a general-purpose LLM with robust in-context machine learning capabilities through continued pretraining with synthesized ML tasks, achieving high performance across various domains without task-specific training.
Large language models in-context learning continued-pretraining structural causal models random-forest teacher token-efficient prompt batch inference out-of-distribution tabular classification many-shot scaling law MMLU