Aligning Visual Foundation Encoders to Tokenizers for Diffusion Models
In this work, we propose aligning pretrained visual encoders to serve as
tokenizers for latent diffusion models in image generation. Unlike training a
variational autoencoder (VAE) from scratch, which primarily emphasizes
low-level details, our approach leverages the rich semantic structure of
foundation encoders. We introduce a three-stage alignment strategy: (1) freeze
the encoder and train an adapter and a decoder to establish a semantic latent
space; (2) jointly optimize all components with an additional semantic
preservation loss, enabling the encoder to capture perceptual details while
retaining high-level semantics; and (3) refine the decoder for improved
reconstruction quality. This alignment yields semantically rich image
tokenizers that benefit diffusion models. On ImageNet 256times256, our
tokenizer accelerates the convergence of diffusion models, reaching a gFID of
1.90 within just 64 epochs, and improves generation both with and without
classifier-free guidance. Scaling to LAION, a 2B-parameter text-to-image model
trained with our tokenizer consistently outperforms FLUX VAE under the same
training steps. Overall, our method is simple, scalable, and establishes a
semantically grounded paradigm for continuous tokenizer design.
68dd5f21ae4a047656f86bc2
Bowei Chen
68dd5f21ae4a047656f86bc3
Sai Bi
68dd5f21ae4a047656f86bc4
Hao Tan
68dd5f21ae4a047656f86bc5
He Zhang
68dd5f21ae4a047656f86bc6
Tianyuan Zhang
68dd5f21ae4a047656f86bc7
Zhengqi Li
68dd5f21ae4a047656f86bc8
Yuanjun Xiong
68dd5f21ae4a047656f86bc9
Jianming Zhang
68dd5f21ae4a047656f86bca
Kai Zhang
Aligning Visual Foundation Encoders to Tokenizers for Diffusion Models
In this work, we propose aligning pretrained visual encoders to serve as
tokenizers for latent diffusion models in image generation. Unlike training a
variational autoencoder (VAE) from scratch, which primarily emphasizes
low-level details, our approach leverages the rich semantic structure of
foundation encoders. We introduce a three-stage alignment strategy: (1) freeze
the encoder and train an adapter and a decoder to establish a semantic latent
space; (2) jointly optimize all components with an additional semantic
preservation loss, enabling the encoder to capture perceptual details while
retaining high-level semantics; and (3) refine the decoder for improved
reconstruction quality. This alignment yields semantically rich image
tokenizers that benefit diffusion models. On ImageNet 256times256, our
tokenizer accelerates the convergence of diffusion models, reaching a gFID of
1.90 within just 64 epochs, and improves generation both with and without
classifier-free guidance. Scaling to LAION, a 2B-parameter text-to-image model
trained with our tokenizer consistently outperforms FLUX VAE under the same
training steps. Overall, our method is simple, scalable, and establishes a
semantically grounded paradigm for continuous tokenizer design.
Pretrained visual encoders are aligned as tokenizers for latent diffusion models, improving image generation quality and convergence speed.
visual encoders tokenizers latent diffusion models semantic latent space adapter decoder semantic preservation loss perceptual details high-level semantics gFID ImageNet LAION FLUX VAE classifier-free guidance