GenTron: Delving Deep into Diffusion Transformers for Image and Video
  Generation
In this study, we explore Transformer-based diffusion models for image and
video generation. Despite the dominance of Transformer architectures in various
fields due to their flexibility and scalability, the visual generative domain
primarily utilizes CNN-based U-Net architectures, particularly in
diffusion-based models. We introduce GenTron, a family of Generative models
employing Transformer-based diffusion, to address this gap. Our initial step
was to adapt Diffusion Transformers (DiTs) from class to text conditioning, a
process involving thorough empirical exploration of the conditioning mechanism.
We then scale GenTron from approximately 900M to over 3B parameters, observing
significant improvements in visual quality. Furthermore, we extend GenTron to
text-to-video generation, incorporating novel motion-free guidance to enhance
video quality. In human evaluations against SDXL, GenTron achieves a 51.1% win
rate in visual quality (with a 19.8% draw rate), and a 42.3% win rate in text
alignment (with a 42.9% draw rate). GenTron also excels in the T2I-CompBench,
underscoring its strengths in compositional generation. We believe this work
will provide meaningful insights and serve as a valuable reference for future
research.
657293329a5c2d6df9f62e52
Shoufa Chen
claimed_verified
2023-12-08T08:00:13.363Z
6412a33900634c4fe9873652
https://cdn-avatars.huggingface.co/v1/production/uploads/6412a33900634c4fe9873652/Nmn_yRA1gGD2VO1YbSOYF.jpeg
Shoufa Chen
user
ShoufaChen
657293329a5c2d6df9f62e53
Mengmeng Xu
claimed_verified
2023-12-08T12:08:31.031Z
6426983737d7e7882786af37
https://cdn-avatars.huggingface.co/v1/production/uploads/6426983737d7e7882786af37/5rG4gJIogEvyXuFahJ9yo.png
Frost XU
user
frostinassiky
657293329a5c2d6df9f62e54
Jiawei Ren
admin_assigned
2023-12-08T10:35:47.970Z
64bf93139a69e8da48146cce
/avatars/0be0719e77312b12b4b3520f98ed808b.svg
Jiawei Ren
user
Jiawei1992
657293329a5c2d6df9f62e55
Yuren Cong
admin_assigned
2023-12-08T10:35:54.417Z
62f8040c47d782a6e286ea8e
/avatars/f45224b851ae79fff9912f6e67159e52.svg
Yuren Cong
user
Yuren
657293329a5c2d6df9f62e56
Sen He
657293329a5c2d6df9f62e57
Yanping Xie
657293329a5c2d6df9f62e58
Animesh Sinha
claimed_verified
2023-12-08T08:00:09.848Z
63772d28e069bdcdbb8e4a29
/avatars/be71b00e009f2093a3ff79f07815e68e.svg
Animesh Sinha
user
animeshsinha
657293329a5c2d6df9f62e59
Ping Luo
657293329a5c2d6df9f62e5a
Tao Xiang
657293329a5c2d6df9f62e5b
Juan-Manuel Perez-Rua
claimed_verified
2023-12-08T11:21:21.728Z
6572f4d86a1344d777fa7535
/avatars/eb19a0d4e8d8019456a78a7d621d0955.svg
Juanma Perez
user
jperezrua
GenTron: Delving Deep into Diffusion Transformers for Image and Video
  Generation
In this study, we explore Transformer-based diffusion models for image and
video generation. Despite the dominance of Transformer architectures in various
fields due to their flexibility and scalability, the visual generative domain
primarily utilizes CNN-based U-Net architectures, particularly in
diffusion-based models. We introduce GenTron, a family of Generative models
employing Transformer-based diffusion, to address this gap. Our initial step
was to adapt Diffusion Transformers (DiTs) from class to text conditioning, a
process involving thorough empirical exploration of the conditioning mechanism.
We then scale GenTron from approximately 900M to over 3B parameters, observing
significant improvements in visual quality. Furthermore, we extend GenTron to
text-to-video generation, incorporating novel motion-free guidance to enhance
video quality. In human evaluations against SDXL, GenTron achieves a 51.1% win
rate in visual quality (with a 19.8% draw rate), and a 42.3% win rate in text
alignment (with a 42.9% draw rate). GenTron also excels in the T2I-CompBench,
underscoring its strengths in compositional generation. We believe this work
will provide meaningful insights and serve as a valuable reference for future
research.
GenTron, a Transformer-based diffusion model family, achieves superior visual quality and text alignment in image and video generation compared to existing methods.
Transformer-based diffusion models GenTron Diffusion Transformers text conditioning parameter scaling text-to-video generation motion-free guidance visual quality text alignment T2I-CompBench compositional generation