Predicting the Order of Upcoming Tokens Improves Language Modeling
Multi-Token Prediction (MTP) has been proposed as an auxiliary objective to
improve next-token prediction (NTP) in language model training but shows
inconsistent improvements, underperforming in standard NLP benchmarks. We argue
that MTP's exact future token prediction is too difficult as an auxiliary loss.
Instead, we propose Token Order Prediction (TOP), which trains models to order
upcoming tokens by their proximity using a learning-to-rank loss. TOP requires
only a single additional unembedding layer compared to MTP's multiple
transformer layers. We pretrain models of 340M, 1.8B, and 7B parameters using
NTP, MTP, and TOP objectives. Results on eight standard NLP benchmarks show
that TOP overall outperforms both NTP and MTP even at scale. Our code is
available at https://github.com/zaydzuhri/token-order-prediction
68afcddc245176306494cc49
Zayd M. K. Zuhri
claimed_verified
2025-08-28T08:54:43.673Z
60cf8a354061635e43b28f60
https://cdn-avatars.huggingface.co/v1/production/uploads/60cf8a354061635e43b28f60/o8gv9mG5cvnopJZJnNibi.jpeg
Zayd Muhammad Kawakibi Zuhri
user
zaydzuhri
68afcddc245176306494cc4a
Erland Hilman Fuadi
claimed_verified
2025-08-28T08:54:41.442Z
6239ac869895e5c2a4345131
https://cdn-avatars.huggingface.co/v1/production/uploads/6239ac869895e5c2a4345131/QErVhX2EdKUSg06eRrU7L.jpeg
Edd
user
Erland
68afcddc245176306494cc4b
Alham Fikri Aji
claimed_verified
2025-08-28T18:17:09.693Z
61a4dc053205e107691e0d82
https://cdn-avatars.huggingface.co/v1/production/uploads/61a4dc053205e107691e0d82/BESoEHlHYXstXudh6dOdT.jpeg
Alham Fikri Aji
user
afaji
Predicting the Order of Upcoming Tokens Improves Language Modeling
Multi-Token Prediction (MTP) has been proposed as an auxiliary objective to
improve next-token prediction (NTP) in language model training but shows
inconsistent improvements, underperforming in standard NLP benchmarks. We argue
that MTP's exact future token prediction is too difficult as an auxiliary loss.
Instead, we propose Token Order Prediction (TOP), which trains models to order
upcoming tokens by their proximity using a learning-to-rank loss. TOP requires
only a single additional unembedding layer compared to MTP's multiple
transformer layers. We pretrain models of 340M, 1.8B, and 7B parameters using
NTP, MTP, and TOP objectives. Results on eight standard NLP benchmarks show
that TOP overall outperforms both NTP and MTP even at scale. Our code is
available at https://github.com/zaydzuhri/token-order-prediction
Token Order Prediction (TOP) improves language model training by ordering upcoming tokens, outperforming both Next-Token Prediction (NTP) and Multi-Token Prediction (MTP) across benchmarks.
Multi-Token Prediction Next-Token Prediction Token Order Prediction learning-to-rank loss unembedding layer transformer layers