Encoder-Decoder Block Diffusion Language Models for Efficient Training and Inference
Marianne Arriola, Yair Schiff, Hao Phung, Aaron Gokaslan, Volodymyr Kuleshov
Discrete diffusion models have steadily improved in quality, but still face a performance gap relative to autoregressive approaches. Partitioning sequences into blocks and performing diffusion within each block improves performance, but doubles computational costs in standard decoder-only architectures. Here, we propose an encoder-decoder architecture for block discrete diffusion, which relies on a lightweight decoder to iteratively refine each block of generated data. This greatly improves training and inference efficiency and allows training larger and higher quality block diffusion models for the same compute budget. We introduce encoder-decoder discrete diffusion (E2D2), a block diffusion model with specialized training and sampling algorithms, and we show that E2D2 achieves superior trade-offs between generation quality and inference throughput on summarization, translation, and mathematical reasoning tasks.