MonoFormer: One Transformer for Both Diffusion and Autoregression
Most existing multimodality methods use separate backbones for
autoregression-based discrete text generation and diffusion-based continuous
visual generation, or the same backbone by discretizing the visual data to use
autoregression for both text and visual generation. In this paper, we propose
to study a simple idea: share one transformer for both autoregression and
diffusion. The feasibility comes from two main aspects: (i) Transformer is
successfully applied to diffusion for visual generation, and (ii) transformer
training for autoregression and diffusion is very similar, and the difference
merely lies in that diffusion uses bidirectional attention mask and
autoregression uses causal attention mask. Experimental results show that our
approach achieves comparable image generation performance to current
state-of-the-art methods as well as maintains the text generation capability.
The project is publicly available at https://monoformer.github.io/.
66f36fd97a2d332b0d68dc5f
Chuyang Zhao
66f36fd97a2d332b0d68dc60
Yuxing Song
66f36fd97a2d332b0d68dc61
Wenhao Wang
claimed_verified
2024-09-26T08:23:31.477Z
62b32a4429a410b7f6b06710
https://cdn-avatars.huggingface.co/v1/production/uploads/62b32a4429a410b7f6b06710/VzgvmnlYZWuifZTkIkCxy.jpeg
Wenhao Wang
user
WenhaoWang
66f36fd97a2d332b0d68dc62
Haocheng Feng
66f36fd97a2d332b0d68dc63
Errui Ding
66f36fd97a2d332b0d68dc64
Yifan Sun
66f36fd97a2d332b0d68dc65
Xinyan Xiao
66f36fd97a2d332b0d68dc66
Jingdong Wang
MonoFormer: One Transformer for Both Diffusion and Autoregression
Most existing multimodality methods use separate backbones for
autoregression-based discrete text generation and diffusion-based continuous
visual generation, or the same backbone by discretizing the visual data to use
autoregression for both text and visual generation. In this paper, we propose
to study a simple idea: share one transformer for both autoregression and
diffusion. The feasibility comes from two main aspects: (i) Transformer is
successfully applied to diffusion for visual generation, and (ii) transformer
training for autoregression and diffusion is very similar, and the difference
merely lies in that diffusion uses bidirectional attention mask and
autoregression uses causal attention mask. Experimental results show that our
approach achieves comparable image generation performance to current
state-of-the-art methods as well as maintains the text generation capability.
The project is publicly available at https://monoformer.github.io/.
A shared transformer backbone for both autoregressive text generation and diffusion-based visual generation achieves comparable performance to state-of-the-art methods.
multimodality methods autoregression diffusion transformer bidirectional attention mask causal attention mask image generation text generation