TESS: Text-to-Text Self-Conditioned Simplex Diffusion
Diffusion models have emerged as a powerful paradigm for generation,
obtaining strong performance in various domains with continuous-valued inputs.
Despite the promises of fully non-autoregressive text generation, applying
diffusion models to natural language remains challenging due to its discrete
nature. In this work, we propose Text-to-text Self-conditioned Simplex
Diffusion (TESS), a text diffusion model that is fully non-autoregressive,
employs a new form of self-conditioning, and applies the diffusion process on
the logit simplex space rather than the typical learned embedding space.
Through extensive experiments on natural language understanding and generation
tasks including summarization, text simplification, paraphrase generation, and
question generation, we demonstrate that TESS outperforms state-of-the-art
non-autoregressive models and is competitive with pretrained autoregressive
sequence-to-sequence models.
6462d5e9d9b4902188119f66
Rabeeh Karimi Mahabadi
6462d5e9d9b4902188119f67
Jaesung Tae
claimed_verified
2023-05-16T15:32:27.927Z
5fca69d8246881afd5b00163
/avatars/028d72c5815d3cb1da3d95828bd5d5ea.svg
Jake Tae
user
jaketae
6462d5e9d9b4902188119f68
Hamish Ivison
claimed_verified
2023-05-16T15:58:10.175Z
62608fc2ffe8827cb1d89f9f
https://cdn-avatars.huggingface.co/v1/production/uploads/1654027835241-62608fc2ffe8827cb1d89f9f.png
Hamish Ivison
user
hamishivi
6462d5e9d9b4902188119f69
James Henderson
6462d5e9d9b4902188119f6a
Iz Beltagy
6462d5e9d9b4902188119f6b
Matthew E. Peters
6462d5e9d9b4902188119f6c
Arman Cohan
TESS: Text-to-Text Self-Conditioned Simplex Diffusion
Diffusion models have emerged as a powerful paradigm for generation,
obtaining strong performance in various domains with continuous-valued inputs.
Despite the promises of fully non-autoregressive text generation, applying
diffusion models to natural language remains challenging due to its discrete
nature. In this work, we propose Text-to-text Self-conditioned Simplex
Diffusion (TESS), a text diffusion model that is fully non-autoregressive,
employs a new form of self-conditioning, and applies the diffusion process on
the logit simplex space rather than the typical learned embedding space.
Through extensive experiments on natural language understanding and generation
tasks including summarization, text simplification, paraphrase generation, and
question generation, we demonstrate that TESS outperforms state-of-the-art
non-autoregressive models and is competitive with pretrained autoregressive
sequence-to-sequence models.
Text-to-text Self-conditioned Simplex Diffusion (TESS) achieves strong performance on natural language tasks by using a fully non-autoregressive approach that applies diffusion in logit simplex space.
diffusion models non-autoregressive text generation self-conditioning logit simplex space summarization text simplification paraphrase generation question generation state-of-the-art non-autoregressive models pretrained autoregressive sequence-to-sequence models