LLM2Vec: Large Language Models Are Secretly Powerful Text Encoders
Large decoder-only language models (LLMs) are the state-of-the-art models on
most of today's NLP tasks and benchmarks. Yet, the community is only slowly
adopting these models for text embedding tasks, which require rich
contextualized representations. In this work, we introduce LLM2Vec, a simple
unsupervised approach that can transform any decoder-only LLM into a strong
text encoder. LLM2Vec consists of three simple steps: 1) enabling bidirectional
attention, 2) masked next token prediction, and 3) unsupervised contrastive
learning. We demonstrate the effectiveness of LLM2Vec by applying it to 3
popular LLMs ranging from 1.3B to 7B parameters and evaluate the transformed
models on English word- and sequence-level tasks. We outperform encoder-only
models by a large margin on word-level tasks and reach a new unsupervised
state-of-the-art performance on the Massive Text Embeddings Benchmark (MTEB).
Moreover, when combining LLM2Vec with supervised contrastive learning, we
achieve state-of-the-art performance on MTEB among models that train only on
publicly available data. Our strong empirical results and extensive analysis
demonstrate that LLMs can be effectively transformed into universal text
encoders in a parameter-efficient manner without the need for expensive
adaptation or synthetic GPT-4 generated data.
6615e08f362219047e082f98
Parishad BehnamGhader
admin_assigned
2024-04-18T14:29:15.707Z
627d5ead401f42c57b6ce54c
https://cdn-avatars.huggingface.co/v1/production/uploads/627d5ead401f42c57b6ce54c/GajmN5G_MRUFRZs6ens0t.jpeg
Parishad BehnamGhader
user
parishadbehnam
6615e08f362219047e082f99
Vaibhav Adlakha
admin_assigned
2024-04-18T14:29:24.085Z
61981af5d420757268e195ac
/avatars/8b59aaf33447224f83d497425fd7ea8f.svg
Vaibhav Adlakha
user
vaibhavad
6615e08f362219047e082f9a
Marius Mosbach
admin_assigned
2024-04-18T14:29:31.079Z
608865d2511e863acdb20bae
https://cdn-avatars.huggingface.co/v1/production/uploads/608865d2511e863acdb20bae/WSJQHZ4PC9hjLMCWv9mSb.png
Marius Mosbach
user
mmosbach
6615e08f362219047e082f9b
Dzmitry Bahdanau
6615e08f362219047e082f9c
Nicolas Chapados
admin_assigned
2024-04-18T14:29:41.309Z
631f54aa5ba8c026340b13cf
https://cdn-avatars.huggingface.co/v1/production/uploads/631f54aa5ba8c026340b13cf/2jI0VUDG5cKkdf2C5KJuy.png
Nicolas Chapados
user
nicolaschapados
6615e08f362219047e082f9d
Siva Reddy
claimed_verified
2024-04-10T10:57:34.527Z
624734dc4c731bb6bfab8af7
/avatars/6b250b58710a3287b85e4733c1824558.svg
Siva Reddy
user
sivareddyg
LLM2Vec: Large Language Models Are Secretly Powerful Text Encoders
Large decoder-only language models (LLMs) are the state-of-the-art models on
most of today's NLP tasks and benchmarks. Yet, the community is only slowly
adopting these models for text embedding tasks, which require rich
contextualized representations. In this work, we introduce LLM2Vec, a simple
unsupervised approach that can transform any decoder-only LLM into a strong
text encoder. LLM2Vec consists of three simple steps: 1) enabling bidirectional
attention, 2) masked next token prediction, and 3) unsupervised contrastive
learning. We demonstrate the effectiveness of LLM2Vec by applying it to 3
popular LLMs ranging from 1.3B to 7B parameters and evaluate the transformed
models on English word- and sequence-level tasks. We outperform encoder-only
models by a large margin on word-level tasks and reach a new unsupervised
state-of-the-art performance on the Massive Text Embeddings Benchmark (MTEB).
Moreover, when combining LLM2Vec with supervised contrastive learning, we
achieve state-of-the-art performance on MTEB among models that train only on
publicly available data. Our strong empirical results and extensive analysis
demonstrate that LLMs can be effectively transformed into universal text
encoders in a parameter-efficient manner without the need for expensive
adaptation or synthetic GPT-4 generated data.
LLM2Vec transforms decoder-only LLMs into effective text encoders using bidirectional attention, masked next token prediction, and unsupervised contrastive learning, achieving state-of-the-art performance on text embedding tasks.
decoder-only language models LLM2Vec bidirectional attention masked next token prediction unsupervised contrastive learning text embedding tasks context large margin performance Massive Text Embeddings Benchmark (MTEB) encoder-only models supervised contrastive learning