ssToken: Self-modulated and Semantic-aware Token Selection for LLM
  Fine-tuning
Data quality plays a critical role in enhancing supervised fine-tuning (SFT)
for large language models (LLMs), and token-level data selection has emerged as
a promising direction for its fine-grained nature. Despite their strong
empirical performance, existing token-level selection methods share two key
limitations: (1) requiring training or accessing an additional reference model,
and (2) relying solely on loss information for token selection, which cannot
well preserve semantically important tokens that are not favored by loss-based
metrics. To address these challenges, we propose ssToken, a Self-modulated and
Semantic-aware Token Selection approach. ssToken leverages readily accessible
history models to compute the per-token loss difference with the current model,
which serves as a self-modulated signal that enables the model to adaptively
select tokens along its optimization trajectory, rather than relying on excess
loss from an offline-trained reference model as in prior works. We further
introduce a semantic-aware, attention-based token importance estimation metric,
orthogonal to loss-based selection and providing complementary semantic
information for more effective filtering. Extensive experiments across
different model families and scales demonstrate that both self-modulated
selection and semantic-aware selection alone outperform full-data fine-tuning,
while their integration--ssToken--achieves synergistic gains and further
surpasses prior token-level selection methods, delivering performance
improvements while maintaining training efficiency.
68f854bc7669bcaeecce0d8b
Xiaohan Qin
claimed_verified
2025-10-22T15:43:27.238Z
64452110e1fd8d65b2790bfc
https://cdn-avatars.huggingface.co/v1/production/uploads/64452110e1fd8d65b2790bfc/3NIsQFV9X44X88aLXR6eE.jpeg
Xiaohan Qin (SJTU) & (SII)
user
galaxy-1
68f854bc7669bcaeecce0d8c
Xiaoxing Wang
68f854bc7669bcaeecce0d8d
Ning Liao
68f854bc7669bcaeecce0d8e
Cancheng Zhang
68f854bc7669bcaeecce0d8f
Xiangdong Zhang
68f854bc7669bcaeecce0d90
Mingquan Feng
68f854bc7669bcaeecce0d91
Jingzhi Wang
68f854bc7669bcaeecce0d92
Junchi Yan
ssToken: Self-modulated and Semantic-aware Token Selection for LLM
  Fine-tuning
Data quality plays a critical role in enhancing supervised fine-tuning (SFT)
for large language models (LLMs), and token-level data selection has emerged as
a promising direction for its fine-grained nature. Despite their strong
empirical performance, existing token-level selection methods share two key
limitations: (1) requiring training or accessing an additional reference model,
and (2) relying solely on loss information for token selection, which cannot
well preserve semantically important tokens that are not favored by loss-based
metrics. To address these challenges, we propose ssToken, a Self-modulated and
Semantic-aware Token Selection approach. ssToken leverages readily accessible
history models to compute the per-token loss difference with the current model,
which serves as a self-modulated signal that enables the model to adaptively
select tokens along its optimization trajectory, rather than relying on excess
loss from an offline-trained reference model as in prior works. We further
introduce a semantic-aware, attention-based token importance estimation metric,
orthogonal to loss-based selection and providing complementary semantic
information for more effective filtering. Extensive experiments across
different model families and scales demonstrate that both self-modulated
selection and semantic-aware selection alone outperform full-data fine-tuning,
while their integration--ssToken--achieves synergistic gains and further
surpasses prior token-level selection methods, delivering performance
improvements while maintaining training efficiency.
ssToken, a self-modulated and semantic-aware token selection approach, enhances supervised fine-tuning of large language models by adaptively selecting tokens and providing complementary semantic information, outperforming existing methods.
supervised fine-tuning large language models token-level selection history models per-token loss difference self-modulated signal semantic-aware attention-based token importance estimation full-data fine-tuning synergistic gains