The Quest for Efficient Reasoning: A Data-Centric Benchmark to CoT
  Distillation
Data-centric distillation, including data augmentation, selection, and
mixing, offers a promising path to creating smaller, more efficient student
Large Language Models (LLMs) that retain strong reasoning abilities. However,
there still lacks a comprehensive benchmark to systematically assess the effect
of each distillation approach. This paper introduces DC-CoT, the first
data-centric benchmark that investigates data manipulation in chain-of-thought
(CoT) distillation from method, model and data perspectives. Utilizing various
teacher models (e.g., o4-mini, Gemini-Pro, Claude-3.5) and student
architectures (e.g., 3B, 7B parameters), we rigorously evaluate the impact of
these data manipulations on student model performance across multiple reasoning
datasets, with a focus on in-distribution (IID) and out-of-distribution (OOD)
generalization, and cross-domain transfer. Our findings aim to provide
actionable insights and establish best practices for optimizing CoT
distillation through data-centric techniques, ultimately facilitating the
development of more accessible and capable reasoning models. The dataset can be
found at https://huggingface.co/datasets/rana-shahroz/DC-COT, while our code is
shared in https://anonymous.4open.science/r/DC-COT-FF4C/.
683551c54f3166e8677b43bb
Ruichen Zhang
683551c54f3166e8677b43bc
Rana Muhammad Shahroz Khan
claimed_verified
2025-05-29T07:44:03.117Z
66c3c6362773cfcc33f214ed
/avatars/f4d99a4165208543f9f4d4f9055b9c6f.svg
Rana Muhammad Shahroz Khan
user
rana-shahroz
683551c54f3166e8677b43bd
Zhen Tan
683551c54f3166e8677b43be
Dawei Li
claimed_verified
2025-05-27T07:49:14.749Z
6474e1afb68461d5cf7c41cc
https://cdn-avatars.huggingface.co/v1/production/uploads/6474e1afb68461d5cf7c41cc/bcoiD_qPrjHUBlB259djg.png
Dawei Li
user
wjldw
683551c54f3166e8677b43bf
Song Wang
683551c54f3166e8677b43c0
Tianlong Chen
The Quest for Efficient Reasoning: A Data-Centric Benchmark to CoT
  Distillation
Data-centric distillation, including data augmentation, selection, and
mixing, offers a promising path to creating smaller, more efficient student
Large Language Models (LLMs) that retain strong reasoning abilities. However,
there still lacks a comprehensive benchmark to systematically assess the effect
of each distillation approach. This paper introduces DC-CoT, the first
data-centric benchmark that investigates data manipulation in chain-of-thought
(CoT) distillation from method, model and data perspectives. Utilizing various
teacher models (e.g., o4-mini, Gemini-Pro, Claude-3.5) and student
architectures (e.g., 3B, 7B parameters), we rigorously evaluate the impact of
these data manipulations on student model performance across multiple reasoning
datasets, with a focus on in-distribution (IID) and out-of-distribution (OOD)
generalization, and cross-domain transfer. Our findings aim to provide
actionable insights and establish best practices for optimizing CoT
distillation through data-centric techniques, ultimately facilitating the
development of more accessible and capable reasoning models. The dataset can be
found at https://huggingface.co/datasets/rana-shahroz/DC-COT, while our code is
shared in https://anonymous.4open.science/r/DC-COT-FF4C/.
DC-CoT provides a comprehensive benchmark for assessing data-centric distillation techniques in chain-of-thought distillation, focusing on performance and generalization across different models and datasets.
data-centric distillation data augmentation data selection data mixing chain-of-thought (CoT) in-distribution (IID) out-of-distribution (OOD) cross-domain transfer