"Principal Components" Enable A New Language of Images
We introduce a novel visual tokenization framework that embeds a provable
PCA-like structure into the latent token space. While existing visual
tokenizers primarily optimize for reconstruction fidelity, they often neglect
the structural properties of the latent space -- a critical factor for both
interpretability and downstream tasks. Our method generates a 1D causal token
sequence for images, where each successive token contributes non-overlapping
information with mathematically guaranteed decreasing explained variance,
analogous to principal component analysis. This structural constraint ensures
the tokenizer extracts the most salient visual features first, with each
subsequent token adding diminishing yet complementary information.
Additionally, we identified and resolved a semantic-spectrum coupling effect
that causes the unwanted entanglement of high-level semantic content and
low-level spectral details in the tokens by leveraging a diffusion decoder.
Experiments demonstrate that our approach achieves state-of-the-art
reconstruction performance and enables better interpretability to align with
the human vision system. Moreover, auto-regressive models trained on our token
sequences achieve performance comparable to current state-of-the-art methods
while requiring fewer tokens for training and inference.
67d0f7032eaba9be7bf76e0e
Xin Wen
claimed_verified
2025-03-12T08:39:00.455Z
63483629ac5172169929da0e
https://cdn-avatars.huggingface.co/v1/production/uploads/1665676793089-noauth.jpeg
Xin Wen
user
xwen99
67d0f7032eaba9be7bf76e0f
Bingchen Zhao
claimed_verified
2025-03-12T08:38:56.945Z
62dcd71075e9787ec5aa41ba
/avatars/f37ce036b76180ed0fa004f9c8c09363.svg
Bingchen Zhao
user
tennant
67d0f7032eaba9be7bf76e10
Ismail Elezi
67d0f7032eaba9be7bf76e11
Jiankang Deng
67d0f7032eaba9be7bf76e12
Xiaojuan Qi
claimed_verified
2025-07-15T19:13:38.415Z
6875266f9cd3191dfddc7071
/avatars/64c581910833b111e9a7bae5b8740229.svg
xiaojuan qi
user
xjqi
"Principal Components" Enable A New Language of Images
We introduce a novel visual tokenization framework that embeds a provable
PCA-like structure into the latent token space. While existing visual
tokenizers primarily optimize for reconstruction fidelity, they often neglect
the structural properties of the latent space -- a critical factor for both
interpretability and downstream tasks. Our method generates a 1D causal token
sequence for images, where each successive token contributes non-overlapping
information with mathematically guaranteed decreasing explained variance,
analogous to principal component analysis. This structural constraint ensures
the tokenizer extracts the most salient visual features first, with each
subsequent token adding diminishing yet complementary information.
Additionally, we identified and resolved a semantic-spectrum coupling effect
that causes the unwanted entanglement of high-level semantic content and
low-level spectral details in the tokens by leveraging a diffusion decoder.
Experiments demonstrate that our approach achieves state-of-the-art
reconstruction performance and enables better interpretability to align with
the human vision system. Moreover, auto-regressive models trained on our token
sequences achieve performance comparable to current state-of-the-art methods
while requiring fewer tokens for training and inference.
A novel visual tokenization framework enhances reconstruction performance and interpretability by embedding a PCA-like structure into the latent token space, resolving semantic-spectrum coupling with a diffusion decoder.
visual tokenization PCA-like structure latent token space token sequence principal component analysis diffusion decoder auto-regressive models