Winning the Pruning Gamble: A Unified Approach to Joint Sample and Token
  Pruning for Efficient Supervised Fine-Tuning
As supervised fine-tuning (SFT) evolves from a lightweight post-training step
into a compute-intensive phase rivaling mid-training in scale, data efficiency
has become critical for aligning large language models (LLMs) under tight
budgets. Existing data pruning methods suffer from a fragmented design: they
operate either at the sample level or the token level in isolation, failing to
jointly optimize both dimensions. This disconnect leads to significant
inefficiencies--high-value samples may still contain redundant tokens, while
token-level pruning often discards crucial instructional or corrective signals
embedded in individual examples. To address this bottleneck, we introduce the
Error-Uncertainty (EU) Plane, a diagnostic framework that jointly characterizes
the heterogeneous utility of training data across samples and tokens. Guided by
this insight, we propose Quadrant-based Tuning (Q-Tuning), a unified framework
that strategically coordinates sample pruning and token pruning. Q-Tuning
employs a two-stage strategy: first, it performs sample-level triage to retain
examples rich in informative misconceptions or calibration signals; second, it
applies an asymmetric token-pruning policy, using a context-aware scoring
mechanism to trim less salient tokens exclusively from misconception samples
while preserving calibration samples in their entirety. Our method sets a new
state of the art across five diverse benchmarks. Remarkably, on SmolLM2-1.7B,
Q-Tuning achieves a +38\% average improvement over the full-data SFT baseline
using only 12.5\% of the original training data. As the first dynamic pruning
approach to consistently outperform full-data training, Q-Tuning provides a
practical and scalable blueprint for maximizing data utilization in
budget-constrained LLM SFT.
68dbf6444159d1f2418f9822
Shaobo Wang
claimed_verified
2025-10-01T10:23:24.707Z
66968099c952e09a4cb29f78
https://cdn-avatars.huggingface.co/v1/production/uploads/66968099c952e09a4cb29f78/n90NI2R3E9_RqCyMjDCQF.webp
Wang
user
Steven-Shaobo
68dbf6444159d1f2418f9823
Jiaming Wang
claimed_verified
2025-10-01T10:23:32.506Z
68355c5ec0003bc40230b3f2
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/WZQ5f8oqpni0i-D3a5R-P.png
jasmineWang
user
Jessamine
68dbf6444159d1f2418f9824
Jiajun Zhang
claimed_verified
2025-10-01T10:23:16.205Z
67bc471acda19f7ad35a2f20
https://cdn-avatars.huggingface.co/v1/production/uploads/67bc471acda19f7ad35a2f20/_dbTYE3f4XF9_OY1qXHhw.jpeg
Jiajun Zhang
user
JiajunZhang
68dbf6444159d1f2418f9825
Cong Wang
68dbf6444159d1f2418f9826
Yue Min
68dbf6444159d1f2418f9827
Zichen Wen
68dbf6444159d1f2418f9828
Fei Huang
68dbf6444159d1f2418f9829
Huiqiang Jiang
68dbf6444159d1f2418f982a
Junyang Lin
68dbf6444159d1f2418f982b
Dayiheng Liu
68dbf6444159d1f2418f982c
Linfeng Zhang
Winning the Pruning Gamble: A Unified Approach to Joint Sample and Token
  Pruning for Efficient Supervised Fine-Tuning
As supervised fine-tuning (SFT) evolves from a lightweight post-training step
into a compute-intensive phase rivaling mid-training in scale, data efficiency
has become critical for aligning large language models (LLMs) under tight
budgets. Existing data pruning methods suffer from a fragmented design: they
operate either at the sample level or the token level in isolation, failing to
jointly optimize both dimensions. This disconnect leads to significant
inefficiencies--high-value samples may still contain redundant tokens, while
token-level pruning often discards crucial instructional or corrective signals
embedded in individual examples. To address this bottleneck, we introduce the
Error-Uncertainty (EU) Plane, a diagnostic framework that jointly characterizes
the heterogeneous utility of training data across samples and tokens. Guided by
this insight, we propose Quadrant-based Tuning (Q-Tuning), a unified framework
that strategically coordinates sample pruning and token pruning. Q-Tuning
employs a two-stage strategy: first, it performs sample-level triage to retain
examples rich in informative misconceptions or calibration signals; second, it
applies an asymmetric token-pruning policy, using a context-aware scoring
mechanism to trim less salient tokens exclusively from misconception samples
while preserving calibration samples in their entirety. Our method sets a new
state of the art across five diverse benchmarks. Remarkably, on SmolLM2-1.7B,
Q-Tuning achieves a +38\% average improvement over the full-data SFT baseline
using only 12.5\% of the original training data. As the first dynamic pruning
approach to consistently outperform full-data training, Q-Tuning provides a
practical and scalable blueprint for maximizing data utilization in
budget-constrained LLM SFT.
Quadrant-based Tuning (Q-Tuning) optimizes both sample and token pruning in supervised fine-tuning of large language models, achieving superior performance with reduced data.
supervised fine-tuning data efficiency large language models data pruning sample-level pruning token-level pruning Error-Uncertainty (EU) Plane Quadrant-based Tuning sample-level triage asymmetric token-pruning context-aware scoring mechanism misconception samples calibration samples dynamic pruning