InfiAlign: A Scalable and Sample-Efficient Framework for Aligning LLMs
  to Enhance Reasoning Capabilities
Large language models (LLMs) have exhibited impressive reasoning abilities on
a wide range of complex tasks. However, enhancing these capabilities through
post-training remains resource intensive, particularly in terms of data and
computational cost. Although recent efforts have sought to improve sample
efficiency through selective data curation, existing methods often rely on
heuristic or task-specific strategies that hinder scalability. In this work, we
introduce InfiAlign, a scalable and sample-efficient post-training framework
that integrates supervised fine-tuning (SFT) with Direct Preference
Optimization (DPO) to align LLMs for enhanced reasoning. At the core of
InfiAlign is a robust data selection pipeline that automatically curates
high-quality alignment data from open-source reasoning datasets using
multidimensional quality metrics. This pipeline enables significant performance
gains while drastically reducing data requirements and remains extensible to
new data sources. When applied to the Qwen2.5-Math-7B-Base model, our SFT model
achieves performance on par with DeepSeek-R1-Distill-Qwen-7B, while using only
approximately 12% of the training data, and demonstrates strong generalization
across diverse reasoning tasks. Additional improvements are obtained through
the application of DPO, with particularly notable gains in mathematical
reasoning tasks. The model achieves an average improvement of 3.89% on AIME
24/25 benchmarks. Our results highlight the effectiveness of combining
principled data selection with full-stage post-training, offering a practical
solution for aligning large reasoning models in a scalable and data-efficient
manner. The model checkpoints are available at
https://huggingface.co/InfiX-ai/InfiAlign-Qwen-7B-SFT.
689594c848b0ae5ca2710e22
Shuo Cai
689594c848b0ae5ca2710e23
Su Lu
689594c848b0ae5ca2710e24
Qi Zhou
689594c848b0ae5ca2710e25
Kejing Yang
689594c848b0ae5ca2710e26
Zhijie Sang
689594c848b0ae5ca2710e27
Congkai Xie
claimed_verified
2025-08-13T07:27:42.893Z
6719f1ad725123d503b5ef3c
https://cdn-avatars.huggingface.co/v1/production/uploads/6719f1ad725123d503b5ef3c/P7yYxNVROsKPOVsqMu0jp.jpeg
Congkai Xie
user
congkai
689594c848b0ae5ca2710e28
Hongxia Yang
InfiAlign: A Scalable and Sample-Efficient Framework for Aligning LLMs
  to Enhance Reasoning Capabilities
Large language models (LLMs) have exhibited impressive reasoning abilities on
a wide range of complex tasks. However, enhancing these capabilities through
post-training remains resource intensive, particularly in terms of data and
computational cost. Although recent efforts have sought to improve sample
efficiency through selective data curation, existing methods often rely on
heuristic or task-specific strategies that hinder scalability. In this work, we
introduce InfiAlign, a scalable and sample-efficient post-training framework
that integrates supervised fine-tuning (SFT) with Direct Preference
Optimization (DPO) to align LLMs for enhanced reasoning. At the core of
InfiAlign is a robust data selection pipeline that automatically curates
high-quality alignment data from open-source reasoning datasets using
multidimensional quality metrics. This pipeline enables significant performance
gains while drastically reducing data requirements and remains extensible to
new data sources. When applied to the Qwen2.5-Math-7B-Base model, our SFT model
achieves performance on par with DeepSeek-R1-Distill-Qwen-7B, while using only
approximately 12% of the training data, and demonstrates strong generalization
across diverse reasoning tasks. Additional improvements are obtained through
the application of DPO, with particularly notable gains in mathematical
reasoning tasks. The model achieves an average improvement of 3.89% on AIME
24/25 benchmarks. Our results highlight the effectiveness of combining
principled data selection with full-stage post-training, offering a practical
solution for aligning large reasoning models in a scalable and data-efficient
manner. The model checkpoints are available at
https://huggingface.co/InfiX-ai/InfiAlign-Qwen-7B-SFT.
InfiAlign, a scalable and sample-efficient post-training framework, combines supervised fine-tuning and Direct Preference Optimization to enhance large language models' reasoning abilities with minimal data and computational cost.
supervised fine-tuning Direct Preference Optimization data selection pipeline multidimensional quality metrics Qwen2.5-Math-7B-Base DeepSeek-R1-Distill-Qwen-7B AIME 24/25 benchmarks