Language-Image Alignment with Fixed Text Encoders
Currently, the most dominant approach to establishing language-image
alignment is to pre-train text and image encoders jointly through contrastive
learning, such as CLIP and its variants. In this work, we question whether such
a costly joint training is necessary. In particular, we investigate if a
pre-trained fixed large language model (LLM) offers a good enough text encoder
to guide visual representation learning. That is, we propose to learn
Language-Image alignment with a Fixed Text encoder (LIFT) from an LLM by
training only the image encoder. Somewhat surprisingly, through comprehensive
benchmarking and ablation studies, we find that this much simplified framework
LIFT is highly effective and it outperforms CLIP in most scenarios that involve
compositional understanding and long captions, while achieving considerable
gains in computational efficiency. Our work takes a first step towards
systematically exploring how text embeddings from LLMs can guide visual
learning and suggests an alternative design choice for learning
language-aligned visual representations.
68413c8eb64ba498925da6a8
Jingfeng Yang
claimed_verified
2025-06-05T07:26:26.842Z
65d45fbf9f087171b805c428
https://cdn-avatars.huggingface.co/v1/production/uploads/65d45fbf9f087171b805c428/bgxwcn2p_D9qEa5vynSxV.jpeg
Jingfeng Yang
user
JingfengY
68413c8eb64ba498925da6a9
Ziyang Wu
claimed_verified
2025-06-06T07:41:57.098Z
64ea89932ca4ff1d53b77548
/avatars/ce3df67ba3ea3197ebf74fbe5e2c0e48.svg
Ziyang Wu
user
robinwuzy
68413c8eb64ba498925da6aa
Yue Zhao
claimed_verified
2025-06-07T13:34:00.155Z
673c1b71223fa8d98458d7ce
/avatars/dd1d71ae55ac4e4460f690f777823605.svg
Yue Zhao
user
zyzk-1337
68413c8eb64ba498925da6ab
Yi Ma
Language-Image Alignment with Fixed Text Encoders
Currently, the most dominant approach to establishing language-image
alignment is to pre-train text and image encoders jointly through contrastive
learning, such as CLIP and its variants. In this work, we question whether such
a costly joint training is necessary. In particular, we investigate if a
pre-trained fixed large language model (LLM) offers a good enough text encoder
to guide visual representation learning. That is, we propose to learn
Language-Image alignment with a Fixed Text encoder (LIFT) from an LLM by
training only the image encoder. Somewhat surprisingly, through comprehensive
benchmarking and ablation studies, we find that this much simplified framework
LIFT is highly effective and it outperforms CLIP in most scenarios that involve
compositional understanding and long captions, while achieving considerable
gains in computational efficiency. Our work takes a first step towards
systematically exploring how text embeddings from LLMs can guide visual
learning and suggests an alternative design choice for learning
language-aligned visual representations.
Learning Language-Image alignment with a Fixed Text encoder (LIFT) using pre-trained large language models effectively guides visual representation learning, outperforming joint training methods like CLIP in compositional understanding and long captions.
contrastive learning CLIP pre-trained fixed large language model LLM Language-Image alignment LIFT image encoder compositional understanding long captions