LongLLaDA: Unlocking Long Context Capabilities in Diffusion LLMs
Large Language Diffusion Models, or diffusion LLMs, have emerged as a
significant focus in NLP research, with substantial effort directed toward
understanding their scalability and downstream task performance. However, their
long-context capabilities remain unexplored, lacking systematic analysis or
methods for context extension. In this work, we present the first systematic
investigation comparing the long-context performance of diffusion LLMs and
traditional auto-regressive LLMs. We first identify a unique characteristic of
diffusion LLMs, unlike auto-regressive LLMs, they maintain remarkably
\textit{stable perplexity} during direct context extrapolation.
Furthermore, where auto-regressive models fail outright during the
Needle-In-A-Haystack task with context exceeding their pretrained length, we
discover diffusion LLMs exhibit a distinct \textit{local perception}
phenomenon, enabling successful retrieval from recent context segments. We
explain both phenomena through the lens of Rotary Position Embedding (RoPE)
scaling theory. Building on these observations, we propose LongLLaDA, a
training-free method that integrates LLaDA with the NTK-based RoPE
extrapolation. Our results validate that established extrapolation scaling laws
remain effective for extending the context windows of diffusion LLMs.
Furthermore, we identify long-context tasks where diffusion LLMs outperform
auto-regressive LLMs and others where they fall short. Consequently, this study
establishes the first context extrapolation method for diffusion LLMs while
providing essential theoretical insights and empirical benchmarks critical for
advancing future research on long-context diffusion LLMs.
68521a9a0164cd131671045c
Xiaoran Liu
claimed_verified
2025-06-18T12:16:22.707Z
64f033ef82c6eea604c4da8b
/avatars/51b93fea7fd68b4274ee03701245dcca.svg
Xiaoran Liu
user
SII-xrliu
68521a9a0164cd131671045d
Zhigeng Liu
68521a9a0164cd131671045e
Zengfeng Huang
68521a9a0164cd131671045f
Qipeng Guo
68521a9a0164cd1316710460
Ziwei He
68521a9a0164cd1316710461
Xipeng Qiu
LongLLaDA: Unlocking Long Context Capabilities in Diffusion LLMs
Large Language Diffusion Models, or diffusion LLMs, have emerged as a
significant focus in NLP research, with substantial effort directed toward
understanding their scalability and downstream task performance. However, their
long-context capabilities remain unexplored, lacking systematic analysis or
methods for context extension. In this work, we present the first systematic
investigation comparing the long-context performance of diffusion LLMs and
traditional auto-regressive LLMs. We first identify a unique characteristic of
diffusion LLMs, unlike auto-regressive LLMs, they maintain remarkably
\textit{stable perplexity} during direct context extrapolation.
Furthermore, where auto-regressive models fail outright during the
Needle-In-A-Haystack task with context exceeding their pretrained length, we
discover diffusion LLMs exhibit a distinct \textit{local perception}
phenomenon, enabling successful retrieval from recent context segments. We
explain both phenomena through the lens of Rotary Position Embedding (RoPE)
scaling theory. Building on these observations, we propose LongLLaDA, a
training-free method that integrates LLaDA with the NTK-based RoPE
extrapolation. Our results validate that established extrapolation scaling laws
remain effective for extending the context windows of diffusion LLMs.
Furthermore, we identify long-context tasks where diffusion LLMs outperform
auto-regressive LLMs and others where they fall short. Consequently, this study
establishes the first context extrapolation method for diffusion LLMs while
providing essential theoretical insights and empirical benchmarks critical for
advancing future research on long-context diffusion LLMs.
This study investigates long-context performance of diffusion LLMs compared to auto-regressive LLMs, identifies their unique characteristics, and proposes LongLLaDA, a training-free method for extending context windows.
diffusion LLMs auto-regressive LLMs stable perplexity local perception Rotary Position Embedding (RoPE) scaling theory LongLLaDA NTK-based RoPE extrapolation context extrapolation scaling laws long-context tasks