MIG: Automatic Data Selection for Instruction Tuning by Maximizing
  Information Gain in Semantic Space
Data quality and diversity are key to the construction of effective
instruction-tuning datasets. % With the increasing availability of open-source
instruction-tuning datasets, it is advantageous to automatically select
high-quality and diverse subsets from a vast amount of data. % Existing methods
typically prioritize instance quality and use heuristic rules to maintain
diversity. % However, this absence of a comprehensive view of the entire
collection often leads to suboptimal results. % Moreover, heuristic rules
generally focus on distance or clustering within the embedding space, which
fails to accurately capture the intent of complex instructions in the semantic
space. % To bridge this gap, we propose a unified method for quantifying the
information content of datasets. This method models the semantic space by
constructing a label graph and quantifies diversity based on the distribution
of information within the graph. % Based on such a measurement, we further
introduce an efficient sampling method that selects data samples iteratively to
Maximize the Information Gain (MIG) in semantic
space. % Experiments on various datasets and base models demonstrate that MIG
consistently outperforms state-of-the-art methods. % Notably, the model
fine-tuned with 5\% Tulu3 data sampled by MIG achieves comparable performance
to the official SFT model trained on the full dataset, with improvements of
+5.73\% on AlpacaEval and +6.89\% on Wildbench.
6805b38355d3c792e1a9d0dd
Yicheng Chen
claimed_verified
2025-04-22T09:59:49.780Z
6752dcb6ec430bad7a26c83e
https://cdn-avatars.huggingface.co/v1/production/uploads/6752dcb6ec430bad7a26c83e/F685Ij72H1gIve4s9pcup.jpeg
Chen Yicheng
user
yichengchen24
6805b38355d3c792e1a9d0de
Yining Li
claimed_verified
2025-04-22T09:59:52.428Z
649988726677f66c2b486392
/avatars/b649a77370660e129726e29504daba34.svg
Yining Li
user
ly015
6805b38355d3c792e1a9d0df
Kai Hu
6805b38355d3c792e1a9d0e0
Zerun Ma
6805b38355d3c792e1a9d0e1
Haochen Ye
6805b38355d3c792e1a9d0e2
Kai Chen
MIG: Automatic Data Selection for Instruction Tuning by Maximizing
  Information Gain in Semantic Space
Data quality and diversity are key to the construction of effective
instruction-tuning datasets. % With the increasing availability of open-source
instruction-tuning datasets, it is advantageous to automatically select
high-quality and diverse subsets from a vast amount of data. % Existing methods
typically prioritize instance quality and use heuristic rules to maintain
diversity. % However, this absence of a comprehensive view of the entire
collection often leads to suboptimal results. % Moreover, heuristic rules
generally focus on distance or clustering within the embedding space, which
fails to accurately capture the intent of complex instructions in the semantic
space. % To bridge this gap, we propose a unified method for quantifying the
information content of datasets. This method models the semantic space by
constructing a label graph and quantifies diversity based on the distribution
of information within the graph. % Based on such a measurement, we further
introduce an efficient sampling method that selects data samples iteratively to
Maximize the Information Gain (MIG) in semantic
space. % Experiments on various datasets and base models demonstrate that MIG
consistently outperforms state-of-the-art methods. % Notably, the model
fine-tuned with 5\% Tulu3 data sampled by MIG achieves comparable performance
to the official SFT model trained on the full dataset, with improvements of
+5.73\% on AlpacaEval and +6.89\% on Wildbench.
A unified method quantifies dataset information content using a label graph and an efficient MIG sampling technique to enhance diversity and quality in instruction-tuning datasets, outperforming existing methods.
label graph information content semantic space MIG (Maximize the Information Gain) sampling method instruction-tuning datasets dataset diversity dataset quality heuristic rules AlpacaEval Wildbench