Sequential Diffusion Language Models
Diffusion language models (DLMs) have strong theoretical efficiency but are
limited by fixed-length decoding and incompatibility with key-value (KV)
caches. Block diffusion mitigates these issues, yet still enforces a fixed
block size and requires expensive training. We introduce Next Sequence
Prediction (NSP), which unifies next-token and next-block prediction, enabling
the model to adaptively determine the generation length at each step. When the
length is fixed to 1, NSP reduces to standard next-token prediction. Building
on NSP, we propose Sequential Diffusion Language Model (SDLM), which can
retrofit pre-trained autoregressive language models (ALMs) at minimal cost.
Specifically, SDLM performs diffusion inference within fixed-size mask blocks,
but dynamically decodes consecutive subsequences based on model confidence,
thereby preserving KV-cache compatibility and improving robustness to varying
uncertainty and semantics across the sequence. Experiments show that SDLM
matches or surpasses strong autoregressive baselines using only 3.5M training
samples, while achieving 2.1 higher throughput than Qwen-2.5. Notably, the
SDLM-32B model delivers even more pronounced efficiency gains, demonstrating
the strong scalability potential of our modeling paradigm. Project page and
codes: https://github.com/OpenGVLab/SDLM
68db4940d2bf1f4b15ec73f9
Yangzhou Liu
claimed_verified
2025-10-03T12:55:18.258Z
66a4a1b722147855b009c9eb
https://cdn-avatars.huggingface.co/v1/production/uploads/66a4a1b722147855b009c9eb/uS6Yxp3Q97U1fiHwh1UVu.jpeg
yangzhou liu
user
lll2343
68db4940d2bf1f4b15ec73fa
Yue Cao
claimed_verified
2025-09-30T06:29:54.229Z
6571382c7644d1128561cebe
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/bnJ-T3k_w1g1Mr7b7LglK.jpeg
Cao Yue
user
yuecao0119
68db4940d2bf1f4b15ec73fb
Hao Li
68db4940d2bf1f4b15ec73fc
Gen Luo
68db4940d2bf1f4b15ec73fd
Zhe Chen
68db4940d2bf1f4b15ec73fe
Weiyun Wang
68db4940d2bf1f4b15ec73ff
Xiaobo Liang
68db4940d2bf1f4b15ec7400
Biqing Qi
68db4940d2bf1f4b15ec7401
Lijun Wu
claimed_verified
2025-10-08T11:49:14.469Z
643e60d96db6ba8c5ee177ad
/avatars/73ac7740e462ba0b53a2f2480d9f1e3e.svg
Lijun Wu
user
apeters
68db4940d2bf1f4b15ec7402
Changyao Tian
claimed_verified
2025-10-10T09:57:53.554Z
64b7475efa7eabaae5f7ba94
https://cdn-avatars.huggingface.co/v1/production/uploads/64b7475efa7eabaae5f7ba94/YLv44PZM6tw1sxACY1-U_.png
Changyao Tian
user
Changyao
68db4940d2bf1f4b15ec7403
Yanting Zhang
68db4940d2bf1f4b15ec7404
Yuqiang Li
68db4940d2bf1f4b15ec7405
Tong Lu
68db4940d2bf1f4b15ec7406
Yu Qiao
68db4940d2bf1f4b15ec7407
Jifeng Dai
68db4940d2bf1f4b15ec7408
Wenhai Wang
Sequential Diffusion Language Models
Diffusion language models (DLMs) have strong theoretical efficiency but are
limited by fixed-length decoding and incompatibility with key-value (KV)
caches. Block diffusion mitigates these issues, yet still enforces a fixed
block size and requires expensive training. We introduce Next Sequence
Prediction (NSP), which unifies next-token and next-block prediction, enabling
the model to adaptively determine the generation length at each step. When the
length is fixed to 1, NSP reduces to standard next-token prediction. Building
on NSP, we propose Sequential Diffusion Language Model (SDLM), which can
retrofit pre-trained autoregressive language models (ALMs) at minimal cost.
Specifically, SDLM performs diffusion inference within fixed-size mask blocks,
but dynamically decodes consecutive subsequences based on model confidence,
thereby preserving KV-cache compatibility and improving robustness to varying
uncertainty and semantics across the sequence. Experiments show that SDLM
matches or surpasses strong autoregressive baselines using only 3.5M training
samples, while achieving 2.1 higher throughput than Qwen-2.5. Notably, the
SDLM-32B model delivers even more pronounced efficiency gains, demonstrating
the strong scalability potential of our modeling paradigm. Project page and
codes: https://github.com/OpenGVLab/SDLM
Sequential Diffusion Language Model (SDLM) enhances pre-trained autoregressive language models by adaptively determining generation length and maintaining KV-cache compatibility, achieving high efficiency and throughput.
diffusion language models fixed-length decoding key-value caches block diffusion next sequence prediction next-token prediction sequential diffusion language model autoregressive language models diffusion inference mask blocks model confidence robustness throughput Qwen-2.5