ProCLIP: Progressive Vision-Language Alignment via LLM-based Embedder
The original CLIP text encoder is limited by a maximum input length of 77
tokens, which hampers its ability to effectively process long texts and perform
fine-grained semantic understanding. In addition, the CLIP text encoder lacks
support for multilingual inputs. All these limitations significantly restrict
its applicability across a broader range of tasks. Recent studies have
attempted to replace the CLIP text encoder with an LLM-based embedder to
enhance its ability in processing long texts, multilingual understanding, and
fine-grained semantic comprehension. However, because the representation spaces
of LLMs and the vision-language space of CLIP are pretrained independently
without alignment priors, direct alignment using contrastive learning can
disrupt the intrinsic vision-language alignment in the CLIP image encoder,
leading to an underutilization of the knowledge acquired during pre-training.
To address this challenge, we propose ProCLIP, a curriculum learning-based
progressive vision-language alignment framework to effectively align the CLIP
image encoder with an LLM-based embedder. Specifically, ProCLIP first distills
knowledge from CLIP's text encoder into the LLM-based embedder to leverage
CLIP's rich pretrained knowledge while establishing initial alignment between
the LLM embedder and CLIP image encoder. Subsequently, ProCLIP further aligns
the CLIP image encoder with the LLM-based embedder through image-text
contrastive tuning, employing self-distillation regularization to avoid
overfitting. To achieve a more effective alignment, instance semantic alignment
loss and embedding structure alignment loss are employed during representation
inheritance and contrastive tuning. The Code is available at
https://github.com/VisionXLab/ProCLIP
68f840da7669bcaeecce0cf0
Xiaoxing Hu
68f840da7669bcaeecce0cf1
Kaicheng Yang
claimed_verified
2025-10-24T16:13:32.884Z
63e202f352b7578dba448ab5
https://cdn-avatars.huggingface.co/v1/production/uploads/63e202f352b7578dba448ab5/8itVBLcv14m7OVsoF8h1o.jpeg
Yang
user
Kaichengalex
68f840da7669bcaeecce0cf2
Ziyong Feng
68f840da7669bcaeecce0cf3
Qi Ming
68f840da7669bcaeecce0cf4
Zonghao Guo
68f840da7669bcaeecce0cf5
Xiang An
68f840da7669bcaeecce0cf6
Ziyong Feng
68f840da7669bcaeecce0cf7
Junchi Yan
68f840da7669bcaeecce0cf8
Xue Yang
ProCLIP: Progressive Vision-Language Alignment via LLM-based Embedder
The original CLIP text encoder is limited by a maximum input length of 77
tokens, which hampers its ability to effectively process long texts and perform
fine-grained semantic understanding. In addition, the CLIP text encoder lacks
support for multilingual inputs. All these limitations significantly restrict
its applicability across a broader range of tasks. Recent studies have
attempted to replace the CLIP text encoder with an LLM-based embedder to
enhance its ability in processing long texts, multilingual understanding, and
fine-grained semantic comprehension. However, because the representation spaces
of LLMs and the vision-language space of CLIP are pretrained independently
without alignment priors, direct alignment using contrastive learning can
disrupt the intrinsic vision-language alignment in the CLIP image encoder,
leading to an underutilization of the knowledge acquired during pre-training.
To address this challenge, we propose ProCLIP, a curriculum learning-based
progressive vision-language alignment framework to effectively align the CLIP
image encoder with an LLM-based embedder. Specifically, ProCLIP first distills
knowledge from CLIP's text encoder into the LLM-based embedder to leverage
CLIP's rich pretrained knowledge while establishing initial alignment between
the LLM embedder and CLIP image encoder. Subsequently, ProCLIP further aligns
the CLIP image encoder with the LLM-based embedder through image-text
contrastive tuning, employing self-distillation regularization to avoid
overfitting. To achieve a more effective alignment, instance semantic alignment
loss and embedding structure alignment loss are employed during representation
inheritance and contrastive tuning. The Code is available at
https://github.com/VisionXLab/ProCLIP
ProCLIP enhances CLIP's text processing capabilities by aligning its image encoder with an LLM-based embedder through curriculum learning and contrastive tuning, preserving CLIP's pretrained knowledge.
CLIP text encoder LLM-based embedder curriculum learning progressive vision-language alignment knowledge distillation image-text contrastive tuning self-distillation regularization instance semantic alignment loss embedding structure alignment loss