Diffusion Beats AR in Data-Constrained Settings
Mihir Prabhudesai, Mengning Wu, Amir Zadeh, Katerina Fragkiadaki, Deepak Pathak
Autoregressive Models (ARMs) have dominated the landscape of large language models (LLMs), resulting in  significant advancements across various tasks. Recently, diffusion-based language models have emerged, demonstrating promising results, yet their advantages over established ARMs remain unclear. In this paper, we investigate the strengths of Masked Diffusion Models (MDMs), showing that while they are computationally less efficient than ARMs, they excel significantly under data-constrained conditions. Through systematic empirical evaluations and detailed scaling law analyses, we illustrate how MDMs leverage repeated data more effectively, achieving superior performance across varying model sizes, unique data volumes, and computational budgets. In comprehensive experiments, including validation loss contours and diverse downstream benchmarks we validate that MDMs consistently yields lower validation losses and tangible practical benefits in data-constrainted settings. Our findings underscore the substantial advantage of diffusion-based approaches in scenarios where data availability is limited, marking them as particularly beneficial in resource-constrained environments.