QLIP: Text-Aligned Visual Tokenization Unifies Auto-Regressive
  Multimodal Understanding and Generation
We introduce Quantized Language-Image Pretraining (QLIP), a visual
tokenization method that combines state-of-the-art reconstruction quality with
state-of-the-art zero-shot image understanding. QLIP trains a
binary-spherical-quantization-based autoencoder with reconstruction and
language-image alignment objectives. We are the first to show that the two
objectives do not need to be at odds. We balance the two loss terms dynamically
during training and show that a two-stage training pipeline effectively mixes
the large-batch requirements of image-language pre-training with the memory
bottleneck imposed by the reconstruction objective. We validate the
effectiveness of QLIP for multimodal understanding and text-conditioned image
generation with a single model. Specifically, QLIP serves as a drop-in
replacement for the visual encoder for LLaVA and the image tokenizer for
LlamaGen with comparable or even better performance. Finally, we demonstrate
that QLIP enables a unified mixed-modality auto-regressive model for
understanding and generation.
67a99dfe98423dca45d8f659
Yue Zhao
claimed_verified
2025-02-10T09:49:43.493Z
638fe91639f7e2a7f9d2a8c6
https://cdn-avatars.huggingface.co/v1/production/uploads/638fe91639f7e2a7f9d2a8c6/hB7DMVODcdAEUdQnXxWA8.jpeg
Yue Zhao
user
zhaoyue-zephyrus
67a99dfe98423dca45d8f65a
Fuzhao Xue
67a99dfe98423dca45d8f65b
Scott Reed
67a99dfe98423dca45d8f65c
Linxi Fan
67a99dfe98423dca45d8f65d
Yuke Zhu
67a99dfe98423dca45d8f65e
Jan Kautz
67a99dfe98423dca45d8f65f
Zhiding Yu
67a99dfe98423dca45d8f660
Philipp Krähenbühl
67a99dfe98423dca45d8f661
De-An Huang
QLIP: Text-Aligned Visual Tokenization Unifies Auto-Regressive
  Multimodal Understanding and Generation
We introduce Quantized Language-Image Pretraining (QLIP), a visual
tokenization method that combines state-of-the-art reconstruction quality with
state-of-the-art zero-shot image understanding. QLIP trains a
binary-spherical-quantization-based autoencoder with reconstruction and
language-image alignment objectives. We are the first to show that the two
objectives do not need to be at odds. We balance the two loss terms dynamically
during training and show that a two-stage training pipeline effectively mixes
the large-batch requirements of image-language pre-training with the memory
bottleneck imposed by the reconstruction objective. We validate the
effectiveness of QLIP for multimodal understanding and text-conditioned image
generation with a single model. Specifically, QLIP serves as a drop-in
replacement for the visual encoder for LLaVA and the image tokenizer for
LlamaGen with comparable or even better performance. Finally, we demonstrate
that QLIP enables a unified mixed-modality auto-regressive model for
understanding and generation.
Quantized Language-Image Pretraining (QLIP) integrates binary-spherical-quantization for image reconstructions and language-image alignment, enhancing multimodal understanding and text-conditioned image generation.
binary-spherical-quantization autoencoder reconstruction language-image alignment multimodal understanding text-conditioned image generation mixed-modality auto-regressive model LLaVA LlamaGen