Not All Correct Answers Are Equal: Why Your Distillation Source Matters
Distillation has emerged as a practical and effective approach to enhance the
reasoning capabilities of open-source language models. In this work, we conduct
a large-scale empirical study on reasoning data distillation by collecting
verified outputs from three state-of-the-art teacher models-AM-Thinking-v1,
Qwen3-235B-A22B, and DeepSeek-R1-on a shared corpus of 1.89 million queries. We
construct three parallel datasets and analyze their distributions, revealing
that AM-Thinking-v1-distilled data exhibits greater token length diversity and
lower perplexity. Student models trained on each dataset are evaluated on
reasoning benchmarks including AIME2024, AIME2025, MATH500, and LiveCodeBench.
The AM-based model consistently achieves the best performance (e.g., 84.3 on
AIME2024, 72.2 on AIME2025, 98.4 on MATH500, and 65.9 on LiveCodeBench) and
demonstrates adaptive output behavior-producing longer responses for harder
tasks and shorter ones for simpler tasks. These findings highlight the value of
high-quality, verified reasoning traces. We release the AM-Thinking-v1 and
Qwen3-235B-A22B distilled datasets to support future research on open and
high-performing reasoning-oriented language models. The datasets are publicly
available on Hugging FaceDatasets are available on Hugging Face:
\href{https://huggingface.co/datasets/a-m-team/AM-Thinking-v1-Distilled{AM-Thinking-v1-Distilled},
https://huggingface.co/datasets/a-m-team/AM-Qwen3-Distilled{AM-Qwen3-Distilled}.}.
682d39e6fa24196dcd10d5e8
Xiaoyu Tian
claimed_verified
2025-05-21T08:40:56.337Z
621499d72be42a56cca7afad
/avatars/3cea70d1a55c8096b4270093c69e4a5e.svg
TianXiaoyu
user
Emperorizzis
682d39e6fa24196dcd10d5e9
Yunjie Ji
682d39e6fa24196dcd10d5ea
Haotian Wang
682d39e6fa24196dcd10d5eb
Shuaiting Chen
682d39e6fa24196dcd10d5ec
Sitong Zhao
682d39e6fa24196dcd10d5ed
Yiping Peng
682d39e6fa24196dcd10d5ee
Han Zhao
682d39e6fa24196dcd10d5ef
Xiangang Li
Not All Correct Answers Are Equal: Why Your Distillation Source Matters
Distillation has emerged as a practical and effective approach to enhance the
reasoning capabilities of open-source language models. In this work, we conduct
a large-scale empirical study on reasoning data distillation by collecting
verified outputs from three state-of-the-art teacher models-AM-Thinking-v1,
Qwen3-235B-A22B, and DeepSeek-R1-on a shared corpus of 1.89 million queries. We
construct three parallel datasets and analyze their distributions, revealing
that AM-Thinking-v1-distilled data exhibits greater token length diversity and
lower perplexity. Student models trained on each dataset are evaluated on
reasoning benchmarks including AIME2024, AIME2025, MATH500, and LiveCodeBench.
The AM-based model consistently achieves the best performance (e.g., 84.3 on
AIME2024, 72.2 on AIME2025, 98.4 on MATH500, and 65.9 on LiveCodeBench) and
demonstrates adaptive output behavior-producing longer responses for harder
tasks and shorter ones for simpler tasks. These findings highlight the value of
high-quality, verified reasoning traces. We release the AM-Thinking-v1 and
Qwen3-235B-A22B distilled datasets to support future research on open and
high-performing reasoning-oriented language models. The datasets are publicly
available on Hugging FaceDatasets are available on Hugging Face:
\href{https://huggingface.co/datasets/a-m-team/AM-Thinking-v1-Distilled{AM-Thinking-v1-Distilled},
https://huggingface.co/datasets/a-m-team/AM-Qwen3-Distilled{AM-Qwen3-Distilled}.}.
Distilling reasoning data from advanced language models improves student model performance across various benchmarks.
distillation reasoning data teacher models AM-Thinking-v1 Qwen3-235B-A22B DeepSeek-R1 token length diversity perplexity student models reasoning benchmarks AIME2024 AIME2025 MATH500 LiveCodeBench adaptive output behavior high-quality verified reasoning traces