Adapting Self-Supervised Representations as a Latent Space for Efficient
  Generation
We introduce Representation Tokenizer (RepTok), a generative modeling
framework that represents an image using a single continuous latent token
obtained from self-supervised vision transformers. Building on a pre-trained
SSL encoder, we fine-tune only the semantic token embedding and pair it with a
generative decoder trained jointly using a standard flow matching objective.
This adaptation enriches the token with low-level, reconstruction-relevant
details, enabling faithful image reconstruction. To preserve the favorable
geometry of the original SSL space, we add a cosine-similarity loss that
regularizes the adapted token, ensuring the latent space remains smooth and
suitable for generation. Our single-token formulation resolves spatial
redundancies of 2D latent spaces and significantly reduces training costs.
Despite its simplicity and efficiency, RepTok achieves competitive results on
class-conditional ImageNet generation and naturally extends to text-to-image
synthesis, reaching competitive zero-shot performance on MS-COCO under
extremely limited training budgets. Our findings highlight the potential of
fine-tuned SSL representations as compact and effective latent spaces for
efficient generative modeling.
68f6a72b24c4489363111672
Ming Gui
claimed_verified
2025-10-21T14:04:22.414Z
6319f120d47392f426cd48be
/avatars/fa6d8761ef7a1a86a8a7e97408dc3142.svg
Ming Gui
user
mgui
68f6a72b24c4489363111673
Johannes Schusterbauer
68f6a72b24c4489363111674
Timy Phan
68f6a72b24c4489363111675
Felix Krause
68f6a72b24c4489363111676
Josh Susskind
68f6a72b24c4489363111677
Miguel Angel Bautista
68f6a72b24c4489363111678
Björn Ommer
Adapting Self-Supervised Representations as a Latent Space for Efficient
  Generation
We introduce Representation Tokenizer (RepTok), a generative modeling
framework that represents an image using a single continuous latent token
obtained from self-supervised vision transformers. Building on a pre-trained
SSL encoder, we fine-tune only the semantic token embedding and pair it with a
generative decoder trained jointly using a standard flow matching objective.
This adaptation enriches the token with low-level, reconstruction-relevant
details, enabling faithful image reconstruction. To preserve the favorable
geometry of the original SSL space, we add a cosine-similarity loss that
regularizes the adapted token, ensuring the latent space remains smooth and
suitable for generation. Our single-token formulation resolves spatial
redundancies of 2D latent spaces and significantly reduces training costs.
Despite its simplicity and efficiency, RepTok achieves competitive results on
class-conditional ImageNet generation and naturally extends to text-to-image
synthesis, reaching competitive zero-shot performance on MS-COCO under
extremely limited training budgets. Our findings highlight the potential of
fine-tuned SSL representations as compact and effective latent spaces for
efficient generative modeling.
A generative modeling framework using a single latent token from self-supervised vision transformers achieves competitive results in image and text-to-image synthesis with reduced training costs.
Representation Tokenizer RepTok generative modeling self-supervised vision transformers SSL encoder semantic token embedding generative decoder flow matching objective cosine-similarity loss latent space class-conditional ImageNet generation text-to-image synthesis MS-COCO zero-shot performance