Learning from the Best, Differently: A Diversity-Driven Rethinking on
  Data Selection
High-quality pre-training data is crutial for large language models, where
quality captures factual reliability and semantic value, and diversity ensures
broad coverage and distributional heterogeneity. Existing approaches typically
rely on single or multiple-dimensional score-based selection. However, directly
selecting top-scored data often degrades performance, and sampling from a
broader range is required to recover results. The above non-monotonicity
between dataset scores and downstream benchmark results reveals a fundamental
bias: score-based methods collapse correlated dimensions, causing top-scored
data to appear high-quality while systematically overlooking diversity. We
argue that ensuring diversity requires decomposing correlated metrics into
orthogonal feature dimensions, from which the top-scored data can be directly
selected. Therefore, we proposed the Orthogonal Diversity-Aware Selection
(ODiS) algorithm, which preserves both quality and diversity during data
selection. First, ODiS evaluates data from multiple dimensions, covering
language quality, knowledge quality, and comprehension difficulty. The
multi-dimensional scores are then decorrelated via Principal Component Analysis
(PCA), yielding orthogonal evaluation dimensions. For each dimension, a
Roberta-based scorer is trained to regress the data onto PCA-projected scores,
enabling scalable inference on large corpora. Finally, ODiS constructs the
training dataset by selecting top-scored data within each orthogonal dimension,
thereby ensuring both quality and diversity. Empirical results show that
ODiS-selected data exhibit less than 2\% inter-dimension overlap, confirming
orthogonality between dimensions. More importantly, models trained with
ODiS-selected data significantly outperform other baselines on downstream
benchmarks, highlighting the necessity of orthogonal, diversity-aware data
selection for LLMs.
68f9b1efb9b2e4ae0467390d
Hongyi He
68f9b1efb9b2e4ae0467390e
Xiao Liu
68f9b1efb9b2e4ae0467390f
Zhenghao Lin
68f9b1efb9b2e4ae04673910
Mingni Tang
68f9b1efb9b2e4ae04673911
Yi Cheng
68f9b1efb9b2e4ae04673912
Jintao Wang
68f9b1efb9b2e4ae04673913
Wenjie Li
68f9b1efb9b2e4ae04673914
Peng Cheng
68f9b1efb9b2e4ae04673915
Yeyun Gong
Learning from the Best, Differently: A Diversity-Driven Rethinking on
  Data Selection
High-quality pre-training data is crutial for large language models, where
quality captures factual reliability and semantic value, and diversity ensures
broad coverage and distributional heterogeneity. Existing approaches typically
rely on single or multiple-dimensional score-based selection. However, directly
selecting top-scored data often degrades performance, and sampling from a
broader range is required to recover results. The above non-monotonicity
between dataset scores and downstream benchmark results reveals a fundamental
bias: score-based methods collapse correlated dimensions, causing top-scored
data to appear high-quality while systematically overlooking diversity. We
argue that ensuring diversity requires decomposing correlated metrics into
orthogonal feature dimensions, from which the top-scored data can be directly
selected. Therefore, we proposed the Orthogonal Diversity-Aware Selection
(ODiS) algorithm, which preserves both quality and diversity during data
selection. First, ODiS evaluates data from multiple dimensions, covering
language quality, knowledge quality, and comprehension difficulty. The
multi-dimensional scores are then decorrelated via Principal Component Analysis
(PCA), yielding orthogonal evaluation dimensions. For each dimension, a
Roberta-based scorer is trained to regress the data onto PCA-projected scores,
enabling scalable inference on large corpora. Finally, ODiS constructs the
training dataset by selecting top-scored data within each orthogonal dimension,
thereby ensuring both quality and diversity. Empirical results show that
ODiS-selected data exhibit less than 2\% inter-dimension overlap, confirming
orthogonality between dimensions. More importantly, models trained with
ODiS-selected data significantly outperform other baselines on downstream
benchmarks, highlighting the necessity of orthogonal, diversity-aware data
selection for LLMs.
The Orthogonal Diversity-Aware Selection (ODiS) algorithm enhances large language model performance by ensuring both quality and diversity in training data through orthogonal decomposition of evaluation dimensions.
Orthogonal Diversity-Aware Selection ODiS Principal Component Analysis PCA Roberta-based scorer language quality knowledge quality comprehension difficulty large language models LLMs