Scaling Diffusion Language Models via Adaptation from Autoregressive
  Models
Diffusion Language Models (DLMs) have emerged as a promising new paradigm for
text generative modeling, potentially addressing limitations of autoregressive
(AR) models. However, current DLMs have been studied at a smaller scale
compared to their AR counterparts and lack fair comparison on language modeling
benchmarks. Additionally, training diffusion models from scratch at scale
remains challenging. Given the prevalence of open-source AR language models, we
propose adapting these models to build text diffusion models. We demonstrate
connections between AR and diffusion modeling objectives and introduce a simple
continual pre-training approach for training diffusion models. Through
systematic evaluation on language modeling, reasoning, and commonsense
benchmarks, we show that we can convert AR models ranging from 127M to 7B
parameters (GPT2 and LLaMA) into diffusion models DiffuGPT and DiffuLLaMA,
using less than 200B tokens for training. Our experimental results reveal that
these models outperform earlier DLMs and are competitive with their AR
counterparts. We release a suite of DLMs (with 127M, 355M, and 7B parameters)
capable of generating fluent text, performing in-context learning, filling in
the middle without prompt re-ordering, and following instructions
https://github.com/HKUNLP/DiffuLLaMA.
6719a80204e2e351e5660f16
Shansan Gong
claimed_verified
2024-10-24T07:15:33.126Z
628c83d186fc004b14e1ed48
/avatars/05ff943a9b89b5f67c5bc254bf45b8f5.svg
Shansan Gong
user
Sansa
6719a80204e2e351e5660f17
Shivam Agarwal
extracted_pending
2024-10-24T01:50:59.734Z
64b83a53cbb0af9bfb106384
/avatars/8fd05154d6c2c17e16f2a5b4272846fc.svg
Shivam Agarwal
user
shivamag99
6719a80204e2e351e5660f18
Yizhe Zhang
6719a80204e2e351e5660f19
Jiacheng Ye
claimed_verified
2025-01-23T09:22:05.255Z
6364cc3912188d67e6526cfa
/avatars/bb5c3200c6d64d1260b7e418cf1a76f9.svg
Carson Ye
user
jiacheng-ye
6719a80204e2e351e5660f1a
Lin Zheng
6719a80204e2e351e5660f1b
Mukai Li
claimed_verified
2024-10-24T07:15:31.268Z
624561c1939c9acec2103534
/avatars/3efdf797c53d153cea7415213fb5afc7.svg
Mukai Li
user
kiaia
6719a80204e2e351e5660f1c
Chenxin An
6719a80204e2e351e5660f1d
Peilin Zhao
6719a80204e2e351e5660f1e
Wei Bi
6719a80204e2e351e5660f1f
Jiawei Han
6719a80204e2e351e5660f20
Hao Peng
6719a80204e2e351e5660f21
Lingpeng Kong
Scaling Diffusion Language Models via Adaptation from Autoregressive
  Models
Diffusion Language Models (DLMs) have emerged as a promising new paradigm for
text generative modeling, potentially addressing limitations of autoregressive
(AR) models. However, current DLMs have been studied at a smaller scale
compared to their AR counterparts and lack fair comparison on language modeling
benchmarks. Additionally, training diffusion models from scratch at scale
remains challenging. Given the prevalence of open-source AR language models, we
propose adapting these models to build text diffusion models. We demonstrate
connections between AR and diffusion modeling objectives and introduce a simple
continual pre-training approach for training diffusion models. Through
systematic evaluation on language modeling, reasoning, and commonsense
benchmarks, we show that we can convert AR models ranging from 127M to 7B
parameters (GPT2 and LLaMA) into diffusion models DiffuGPT and DiffuLLaMA,
using less than 200B tokens for training. Our experimental results reveal that
these models outperform earlier DLMs and are competitive with their AR
counterparts. We release a suite of DLMs (with 127M, 355M, and 7B parameters)
capable of generating fluent text, performing in-context learning, filling in
the middle without prompt re-ordering, and following instructions
https://github.com/HKUNLP/DiffuLLaMA.
Converted autoregressive language models into diffusion language models, demonstrating competitive performance on language modeling and reasoning tasks using fewer training tokens.
Diffusion Language Models DLMs autoregressive models language modeling benchmarks diffusion modeling objectives continual pre-training text diffusion models in-context learning DiffuGPT DiffuLLaMA