Direct Multi-Token Decoding
Decoder-only transformers have become the standard architecture for large
language models (LLMs) due to their strong performance. Recent studies suggest
that, in pre-trained LLMs, early, middle, and late layers may serve distinct
roles: Early layers focus on understanding the input context, middle layers
handle task-specific processing, and late layers convert abstract
representations into output tokens. We hypothesize that once representations
have been processed by the early and middle layers, the resulting hidden states
may encapsulate sufficient information to support the generation of multiple
tokens using only the late layers, eliminating the need to repeatedly traverse
the early and middle layers. We refer to this inference paradigm as Direct
Multi-Token Decoding (DMTD). Unlike speculative decoding, our method introduces
no additional parameters, auxiliary routines, or post-generation verification.
Despite being trained on a limited dataset, a fine-tuned DMTD Qwen3-4B model
has already demonstrated promising results, achieving up to a 2x speedup with
only minor performance loss. Moreover, as shown in our scaling analysis, its
performance is expected to further improve with larger training datasets.
68f0539136f8b025381e1915
Xuan Luo
68f0539136f8b025381e1916
Weizhi Wang
claimed_verified
2025-10-16T06:38:58.035Z
63d34004b734eaa4d4faeccf
https://cdn-avatars.huggingface.co/v1/production/uploads/63d34004b734eaa4d4faeccf/zf6d1p0GN8gsagi8N6y4V.jpeg
Weizhi Wang
user
weizhiwang
68f0539136f8b025381e1917
Xifeng Yan
Direct Multi-Token Decoding
Decoder-only transformers have become the standard architecture for large
language models (LLMs) due to their strong performance. Recent studies suggest
that, in pre-trained LLMs, early, middle, and late layers may serve distinct
roles: Early layers focus on understanding the input context, middle layers
handle task-specific processing, and late layers convert abstract
representations into output tokens. We hypothesize that once representations
have been processed by the early and middle layers, the resulting hidden states
may encapsulate sufficient information to support the generation of multiple
tokens using only the late layers, eliminating the need to repeatedly traverse
the early and middle layers. We refer to this inference paradigm as Direct
Multi-Token Decoding (DMTD). Unlike speculative decoding, our method introduces
no additional parameters, auxiliary routines, or post-generation verification.
Despite being trained on a limited dataset, a fine-tuned DMTD Qwen3-4B model
has already demonstrated promising results, achieving up to a 2x speedup with
only minor performance loss. Moreover, as shown in our scaling analysis, its
performance is expected to further improve with larger training datasets.
Direct Multi-Token Decoding (DMTD) accelerates large language model inference by using only late layers for token generation, achieving significant speedup with minimal performance loss.
decoder-only transformers large language models (LLMs) early layers middle layers late layers hidden states Direct Multi-Token Decoding (DMTD) speculative decoding fine-tuned Qwen3-4B scaling analysis