Generative Representational Instruction Tuning
All text-based language problems can be reduced to either generation or
embedding. Current models only perform well at one or the other. We introduce
generative representational instruction tuning (GRIT) whereby a large language
model is trained to handle both generative and embedding tasks by
distinguishing between them through instructions. Compared to other open
models, our resulting GritLM 7B sets a new state of the art on the Massive Text
Embedding Benchmark (MTEB) and outperforms all models up to its size on a range
of generative tasks. By scaling up further, GritLM 8x7B outperforms all open
generative language models that we tried while still being among the best
embedding models. Notably, we find that GRIT matches training on only
generative or embedding data, thus we can unify both at no performance loss.
Among other benefits, the unification via GRIT speeds up Retrieval-Augmented
Generation (RAG) by > 60% for long documents, by no longer requiring separate
retrieval and generation models. Models, code, etc. are freely available at
https://github.com/ContextualAI/gritlm.
65cec094e1c8a3d33cf694d7
Niklas Muennighoff
admin_assigned
2024-02-16T10:30:18.105Z
5f1eb362eec0ad2a071ad6e2
https://cdn-avatars.huggingface.co/v1/production/uploads/5f1eb362eec0ad2a071ad6e2/IXMYkYKuTwn6kBdWnQeeY.png
Niklas Muennighoff
user
Muennighoff
65cec094e1c8a3d33cf694d8
Hongjin Su
claimed_verified
2024-07-19T11:25:09.019Z
632d53951538d4798a73c849
/avatars/b7d0a895e669bcd1303c4716b5401c36.svg
Hongjin SU
user
multi-train
65cec094e1c8a3d33cf694d9
Liang Wang
65cec094e1c8a3d33cf694da
Nan Yang
claimed_verified
2024-05-13T07:49:59.502Z
6341342a9948f573f36d77da
/avatars/813717cb180b2ce566421b43dadabb43.svg
nyanyanya
user
nyanyanya
65cec094e1c8a3d33cf694db
Furu Wei
admin_assigned
2024-02-16T10:34:15.306Z
6368c512fbfe97c16a40baba
/avatars/1c23bc7c0b6d9225699ce27647623d7a.svg
Furu Wei
user
thegenerality
65cec094e1c8a3d33cf694dc
Tao Yu
65cec094e1c8a3d33cf694dd
Amanpreet Singh
claimed_verified
2024-02-19T09:34:42.650Z
6230c6ecfd8b720a5648f6c4
https://cdn-avatars.huggingface.co/v1/production/uploads/1652253065548-6230c6ecfd8b720a5648f6c4.jpeg
Amanpreet Singh
user
aps
65cec094e1c8a3d33cf694de
Douwe Kiela
admin_assigned
2024-02-16T10:34:07.576Z
61dc997715b47073db1620dc
https://cdn-avatars.huggingface.co/v1/production/uploads/1641847245435-61dc997715b47073db1620dc.jpeg
Douwe Kiela
user
douwekiela
Generative Representational Instruction Tuning
All text-based language problems can be reduced to either generation or
embedding. Current models only perform well at one or the other. We introduce
generative representational instruction tuning (GRIT) whereby a large language
model is trained to handle both generative and embedding tasks by
distinguishing between them through instructions. Compared to other open
models, our resulting GritLM 7B sets a new state of the art on the Massive Text
Embedding Benchmark (MTEB) and outperforms all models up to its size on a range
of generative tasks. By scaling up further, GritLM 8x7B outperforms all open
generative language models that we tried while still being among the best
embedding models. Notably, we find that GRIT matches training on only
generative or embedding data, thus we can unify both at no performance loss.
Among other benefits, the unification via GRIT speeds up Retrieval-Augmented
Generation (RAG) by > 60% for long documents, by no longer requiring separate
retrieval and generation models. Models, code, etc. are freely available at
https://github.com/ContextualAI/gritlm.
GrIT allows large language models to excel at both generative and embedding tasks through task instruction, leading to new state-of-the-art performance without sacrificing efficiency.
generative representational instruction tuning GRIT GritLM Massive Text Embedding Benchmark MTEB generative language models embedding models Retrieval-Augmented Generation RAG