Multi-Token Prediction Needs Registers
Multi-token prediction has emerged as a promising objective for improving
language model pretraining, but its benefits have not consistently generalized
to other settings such as fine-tuning. In this paper, we propose MuToR, a
simple and effective approach to multi-token prediction that interleaves
learnable register tokens into the input sequence, each tasked with predicting
future targets. Compared to existing methods, MuToR offers several key
advantages: it introduces only a negligible number of additional parameters,
requires no architectural changes--ensuring compatibility with off-the-shelf
pretrained language models--and remains aligned with the next-token pretraining
objective, making it especially well-suited for supervised fine-tuning.
Moreover, it naturally supports scalable prediction horizons. We demonstrate
the effectiveness and versatility of MuToR across a range of use cases,
including supervised fine-tuning, parameter-efficient fine-tuning (PEFT), and
pretraining, on challenging generative tasks in both language and vision
domains. Our code will be available at: https://github.com/nasosger/MuToR.
68271c682f2e31ef0667bfaf
Anastasios Gerontopoulos
claimed_verified
2025-05-18T19:39:25.735Z
668e4d1b446c8736208d99e1
/avatars/dbe10f3b181e789d98b9b6bde4f711b2.svg
Anastasios Gerontopoulos
user
nasos10
68271c682f2e31ef0667bfb0
Spyros Gidaris
68271c682f2e31ef0667bfb1
Nikos Komodakis
Multi-Token Prediction Needs Registers
Multi-token prediction has emerged as a promising objective for improving
language model pretraining, but its benefits have not consistently generalized
to other settings such as fine-tuning. In this paper, we propose MuToR, a
simple and effective approach to multi-token prediction that interleaves
learnable register tokens into the input sequence, each tasked with predicting
future targets. Compared to existing methods, MuToR offers several key
advantages: it introduces only a negligible number of additional parameters,
requires no architectural changes--ensuring compatibility with off-the-shelf
pretrained language models--and remains aligned with the next-token pretraining
objective, making it especially well-suited for supervised fine-tuning.
Moreover, it naturally supports scalable prediction horizons. We demonstrate
the effectiveness and versatility of MuToR across a range of use cases,
including supervised fine-tuning, parameter-efficient fine-tuning (PEFT), and
pretraining, on challenging generative tasks in both language and vision
domains. Our code will be available at: https://github.com/nasosger/MuToR.
MuToR is a multi-token prediction approach that integrates learnable register tokens to future target prediction, enhancing language model pretraining and fine-tuning without significant parameter increase or architectural changes.
multi-token prediction learnable register tokens fine-tuning parameter-efficient fine-tuning pretraining next-token pretraining objective generative tasks