Winning the Pruning Gamble: A Unified Approach to Joint Sample and Token
  Pruning for Efficient Supervised Fine-Tuning
As supervised fine-tuning (SFT) evolves from a lightweight post-training step
into a compute-intensive phase rivaling mid-training in scale, data efficiency
has become critical for aligning large language models (LLMs) under tight
budgets. Existing data pruning methods suffer from a fragmented design: they
operate either at the sample level or the token level in isolation, failing to
jointly optimize both dimensions. This disconnect leads to significant
inefficiencies--high-value samples may still contain redundant tokens, while
token-level pruning often discards crucial instructional or corrective signals
embedded in individual examples. To address this bottleneck, we introduce the
Error-Uncertainty (EU) Plane, a diagnostic framework that jointly characterizes
the heterogeneous utility of training data across samples and tokens. Guided by
this insight, we propose Quadrant-based Tuning (Q-Tuning), a unified framework
that strategically coordinates sample pruning and token pruning. Q-Tuning
employs a two-stage strategy: first, it performs sample-level triage to retain
examples rich in informative misconceptions or calibration signals; second, it
applies an asymmetric token-pruning policy, using a context-aware scoring
mechanism to trim less salient tokens exclusively from misconception samples
while preserving calibration samples in their entirety. Our method sets a new
state of the art across five diverse benchmarks. Remarkably, on SmolLM2-1.7B,
Q-Tuning achieves a +38\% average improvement over the full-data SFT baseline
using only 12.5\% of the original training data. As the first dynamic pruning
approach to consistently outperform full-data training, Q-Tuning provides a
practical and scalable blueprint for maximizing data utilization in
budget-constrained LLM SFT.
68dbf6444159d1f2418f9822
Shaobo Wang
claimed_verified
2025-10-01T10:23:24.707Z
66968099c952e09a4cb29f78
https://cdn-avatars.huggingface.co/v1/production/uploads/66968099c952e09a4cb29f78/n90NI2R3E9_RqCyMjDCQF.webp
Wang
user
Steven-Shaobo
68dbf6444159d1f2418f9823
Jiaming Wang
claimed_verified
2025-10-01T10:23:32.506Z
68355c5ec0003bc40230b3f2
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/WZQ5f8oqpni0i-D3a5R-P.png
jasmineWang
user
Jessamine
68dbf6444159d1f2418f9824
Jiajun Zhang
claimed_verified
2025-10-01T10:23:16.205Z
67bc471acda19f7ad35a2f20
https://cdn-avatars.huggingface.co/v1/production/uploads/67bc471acda19f7ad35a2f20/_dbTYE3f4XF9_OY1qXHhw.jpeg
Jiajun Zhang
user
JiajunZhang
68dbf6444159d1f2418f9825
Cong Wang
claimed_verified
2025-10-15T07:08:20.125Z
64e48a4594d56c7ee216d0b8
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/y0KL4YCPjMwqvfCs6HdaO.jpeg
王一
user
SongWWWWWW
68dbf6444159d1f2418f9826
Yue Min
68dbf6444159d1f2418f9827
Zichen Wen
claimed_verified
2025-10-20T12:14:33.046Z
653b8c3e97a4d71d950e2f20
/avatars/b68880022e14556d0be58c69615db3be.svg
Zichen Wen
user
zichenwen
68dbf6444159d1f2418f9828
Fei Huang
68dbf6444159d1f2418f9829
Huiqiang Jiang
68dbf6444159d1f2418f982a
Junyang Lin
68dbf6444159d1f2418f982b
Dayiheng Liu
68dbf6444159d1f2418f982c
Linfeng Zhang
Winning the Pruning Gamble: A Unified Approach to Joint Sample and Token
  Pruning for Efficient Supervised Fine-Tuning
As supervised fine-tuning (SFT) evolves from a lightweight post-training step
into a compute-intensive phase rivaling mid-training in scale, data efficiency
has become critical for aligning large language models (LLMs) under tight
budgets. Existing data pruning methods suffer from a fragmented design: they
operate either at the sample level or the token level in isolation, failing to
jointly optimize both dimensions. This disconnect leads to significant
inefficiencies--high-value samples may still contain redundant tokens, while
token-level pruning often discards crucial instructional or corrective signals
embedded in individual examples. To address this bottleneck, we introduce the
Error-Uncertainty (EU) Plane, a diagnostic framework that jointly characterizes
the heterogeneous utility of training data across samples and tokens. Guided by
this insight, we propose Quadrant-based Tuning (Q-Tuning), a unified framework
that strategically coordinates sample pruning and token pruning. Q-Tuning
employs a two-stage strategy: first, it performs sample-level triage to retain
examples rich in informative misconceptions or calibration signals; second, it
applies an asymmetric token-pruning policy, using a context-aware scoring
mechanism to trim less salient tokens exclusively from misconception samples
while preserving calibration samples in their entirety. Our method sets a new
state of the art across five diverse benchmarks. Remarkably, on SmolLM2-1.7B,
Q-Tuning achieves a +38\% average improvement over the full-data SFT baseline
using only 12.5\% of the original training data. As the first dynamic pruning
approach to consistently outperform full-data training, Q-Tuning provides a
practical and scalable blueprint for maximizing data utilization in
budget-constrained LLM SFT.
Quadrant-based Tuning (Q-Tuning) optimizes both sample and token pruning in supervised fine-tuning of large language models, achieving superior performance with reduced data.
supervised fine-tuning data efficiency large language models data pruning sample-level pruning token-level pruning Error-Uncertainty (EU) Plane Quadrant-based Tuning sample-level triage asymmetric token-pruning context-aware scoring mechanism misconception samples calibration samples dynamic pruning