UT5: Pretraining Non autoregressive T5 with unrolled denoising
Recent advances in Transformer-based Large Language Models have made great
strides in natural language generation. However, to decode K tokens, an
autoregressive model needs K sequential forward passes, which may be a
performance bottleneck for large language models. Many non-autoregressive (NAR)
research are aiming to address this sequentiality bottleneck, albeit many have
focused on a dedicated architecture in supervised benchmarks. In this work, we
studied unsupervised pretraining for non auto-regressive T5 models via unrolled
denoising and shown its SoTA results in downstream generation tasks such as
SQuAD question generation and XSum.
6555a8a5c9d29746224c22b5
Mahmoud G. Salem
6555a8a5c9d29746224c22b6
Jiayu Ye
6555a8a5c9d29746224c22b7
Chu-Cheng Lin
admin_assigned
2023-11-16T10:26:08.528Z
6137b1b20317b0a5c10877c6
/avatars/8eccd696bffab67e431aed200e96af14.svg
Chu-Cheng Lin
user
kitsing
6555a8a5c9d29746224c22b8
Frederick Liu
UT5: Pretraining Non autoregressive T5 with unrolled denoising
Recent advances in Transformer-based Large Language Models have made great
strides in natural language generation. However, to decode K tokens, an
autoregressive model needs K sequential forward passes, which may be a
performance bottleneck for large language models. Many non-autoregressive (NAR)
research are aiming to address this sequentiality bottleneck, albeit many have
focused on a dedicated architecture in supervised benchmarks. In this work, we
studied unsupervised pretraining for non auto-regressive T5 models via unrolled
denoising and shown its SoTA results in downstream generation tasks such as
SQuAD question generation and XSum.
Unsupervised pretraining of non-autoregressive T5 models using unrolled denoising achieves state-of-the-art results in downstream generation tasks.
Transformer-based Large Language Models autoregressive models non-autoregressive (NAR) models T5 models unrolled denoising SoTA results SQuAD question generation XSum