Scalable GANs with Transformers
Scalability has driven recent advances in generative modeling, yet its
principles remain underexplored for adversarial learning. We investigate the
scalability of Generative Adversarial Networks (GANs) through two design
choices that have proven to be effective in other types of generative models:
training in a compact Variational Autoencoder latent space and adopting purely
transformer-based generators and discriminators. Training in latent space
enables efficient computation while preserving perceptual fidelity, and this
efficiency pairs naturally with plain transformers, whose performance scales
with computational budget. Building on these choices, we analyze failure modes
that emerge when naively scaling GANs. Specifically, we find issues as
underutilization of early layers in the generator and optimization instability
as the network scales. Accordingly, we provide simple and scale-friendly
solutions as lightweight intermediate supervision and width-aware learning-rate
adjustment. Our experiments show that GAT, a purely transformer-based and
latent-space GANs, can be easily trained reliably across a wide range of
capacities (S through XL). Moreover, GAT-XL/2 achieves state-of-the-art
single-step, class-conditional generation performance (FID of 2.96) on
ImageNet-256 in just 40 epochs, 6x fewer epochs than strong baselines.
68dbabd2d2bf1f4b15ec795e
Sangeek Hyun
claimed_verified
2025-09-30T10:17:25.252Z
6359305490a313828b9ed928
/avatars/a41b3b11922234f4ff268bbd26255a8f.svg
Sangeek Hyun
user
hsi1032
68dbabd2d2bf1f4b15ec795f
MinKyu Lee
68dbabd2d2bf1f4b15ec7960
Jae-Pil Heo
Scalable GANs with Transformers
Scalability has driven recent advances in generative modeling, yet its
principles remain underexplored for adversarial learning. We investigate the
scalability of Generative Adversarial Networks (GANs) through two design
choices that have proven to be effective in other types of generative models:
training in a compact Variational Autoencoder latent space and adopting purely
transformer-based generators and discriminators. Training in latent space
enables efficient computation while preserving perceptual fidelity, and this
efficiency pairs naturally with plain transformers, whose performance scales
with computational budget. Building on these choices, we analyze failure modes
that emerge when naively scaling GANs. Specifically, we find issues as
underutilization of early layers in the generator and optimization instability
as the network scales. Accordingly, we provide simple and scale-friendly
solutions as lightweight intermediate supervision and width-aware learning-rate
adjustment. Our experiments show that GAT, a purely transformer-based and
latent-space GANs, can be easily trained reliably across a wide range of
capacities (S through XL). Moreover, GAT-XL/2 achieves state-of-the-art
single-step, class-conditional generation performance (FID of 2.96) on
ImageNet-256 in just 40 epochs, 6x fewer epochs than strong baselines.
Transformers-based GANs trained in latent space achieve state-of-the-art performance with efficient scaling and reliable training across various capacities.
Generative Adversarial Networks GANs Variational Autoencoder latent space transformers perceptual fidelity intermediate supervision width-aware learning-rate adjustment GAT FID ImageNet-256