Latent Denoising Makes Good Visual Tokenizers
Despite their fundamental role, it remains unclear what properties could make
visual tokenizers more effective for generative modeling. We observe that
modern generative models share a conceptually similar training objective --
reconstructing clean signals from corrupted inputs such as Gaussian noise or
masking -- a process we term denoising. Motivated by this insight, we propose
aligning tokenizer embeddings directly with the downstream denoising objective,
encouraging latent embeddings to be more easily reconstructed even when heavily
corrupted. To achieve this, we introduce the Latent Denoising Tokenizer
(l-DeTok), a simple yet effective tokenizer trained to reconstruct clean images
from latent embeddings corrupted by interpolative noise and random masking.
Extensive experiments on ImageNet 256x256 demonstrate that our tokenizer
consistently outperforms standard tokenizers across six representative
generative models. Our findings highlight denoising as a fundamental design
principle for tokenizer development, and we hope it could motivate new
perspectives for future tokenizer design.
687ef04a33947f780d9b4a71
Jiawei Yang
687ef04a33947f780d9b4a72
Tianhong Li
687ef04a33947f780d9b4a73
Lijie Fan
687ef04a33947f780d9b4a74
Yonglong Tian
687ef04a33947f780d9b4a75
Yue Wang
Latent Denoising Makes Good Visual Tokenizers
Despite their fundamental role, it remains unclear what properties could make
visual tokenizers more effective for generative modeling. We observe that
modern generative models share a conceptually similar training objective --
reconstructing clean signals from corrupted inputs such as Gaussian noise or
masking -- a process we term denoising. Motivated by this insight, we propose
aligning tokenizer embeddings directly with the downstream denoising objective,
encouraging latent embeddings to be more easily reconstructed even when heavily
corrupted. To achieve this, we introduce the Latent Denoising Tokenizer
(l-DeTok), a simple yet effective tokenizer trained to reconstruct clean images
from latent embeddings corrupted by interpolative noise and random masking.
Extensive experiments on ImageNet 256x256 demonstrate that our tokenizer
consistently outperforms standard tokenizers across six representative
generative models. Our findings highlight denoising as a fundamental design
principle for tokenizer development, and we hope it could motivate new
perspectives for future tokenizer design.
Latent Denoising Tokenizer (l-DeTok) improves generative modeling by aligning tokenizer embeddings with a denoising objective, outperforming standard tokenizers across various models.
denoising tokenizer embeddings latent embeddings interpolative noise random masking Latent Denoising Tokenizer l-DeTok generative models ImageNet 256x256