Gecko: Versatile Text Embeddings Distilled from Large Language Models
We present Gecko, a compact and versatile text embedding model. Gecko
achieves strong retrieval performance by leveraging a key idea: distilling
knowledge from large language models (LLMs) into a retriever. Our two-step
distillation process begins with generating diverse, synthetic paired data
using an LLM. Next, we further refine the data quality by retrieving a set of
candidate passages for each query, and relabeling the positive and hard
negative passages using the same LLM. The effectiveness of our approach is
demonstrated by the compactness of the Gecko. On the Massive Text Embedding
Benchmark (MTEB), Gecko with 256 embedding dimensions outperforms all existing
entries with 768 embedding size. Gecko with 768 embedding dimensions achieves
an average score of 66.31, competing with 7x larger models and 5x higher
dimensional embeddings.
660a3edbdf88d3dbac12b9e2
Jinhyuk Lee
extracted_confirmed
2024-04-09T18:33:23.443Z
63fd3edc3c880680af44aa78
/avatars/98759e23e89f9da3ce13266d030e611b.svg
Jinhyuk Lee
user
jinhyuklee
660a3edbdf88d3dbac12b9e3
Zhuyun Dai
660a3edbdf88d3dbac12b9e4
Xiaoqi Ren
660a3edbdf88d3dbac12b9e5
Blair Chen
660a3edbdf88d3dbac12b9e6
Daniel Cer
660a3edbdf88d3dbac12b9e7
Jeremy R. Cole
660a3edbdf88d3dbac12b9e8
Kai Hui
660a3edbdf88d3dbac12b9e9
Michael Boratko
660a3edbdf88d3dbac12b9ea
Rajvi Kapadia
660a3edbdf88d3dbac12b9eb
Wen Ding
660a3edbdf88d3dbac12b9ec
Yi Luan
660a3edbdf88d3dbac12b9ed
Sai Meher Karthik Duddu
660a3edbdf88d3dbac12b9ee
Gustavo Hernandez Abrego
660a3edbdf88d3dbac12b9ef
Weiqiang Shi
660a3edbdf88d3dbac12b9f0
Nithi Gupta
660a3edbdf88d3dbac12b9f1
Aditya Kusupati
claimed_verified
2024-10-16T09:45:24.961Z
65dd3b03f0f26b277be1e91e
/avatars/67160c2d5241aea5c5d9b211f21f9caa.svg
Aditya Kusupati
user
adityakusupati
660a3edbdf88d3dbac12b9f2
Prateek Jain
660a3edbdf88d3dbac12b9f3
Siddhartha Reddy Jonnalagadda
660a3edbdf88d3dbac12b9f4
Ming-Wei Chang
660a3edbdf88d3dbac12b9f5
Iftekhar Naim
Gecko: Versatile Text Embeddings Distilled from Large Language Models
We present Gecko, a compact and versatile text embedding model. Gecko
achieves strong retrieval performance by leveraging a key idea: distilling
knowledge from large language models (LLMs) into a retriever. Our two-step
distillation process begins with generating diverse, synthetic paired data
using an LLM. Next, we further refine the data quality by retrieving a set of
candidate passages for each query, and relabeling the positive and hard
negative passages using the same LLM. The effectiveness of our approach is
demonstrated by the compactness of the Gecko. On the Massive Text Embedding
Benchmark (MTEB), Gecko with 256 embedding dimensions outperforms all existing
entries with 768 embedding size. Gecko with 768 embedding dimensions achieves
an average score of 66.31, competing with 7x larger models and 5x higher
dimensional embeddings.
Gecko is a compact and versatile text embedding model that distills knowledge from large language models to achieve strong retrieval performance on benchmarks.
text embedding model distillation large language models synthetic paired data candidate passages retriever hard negative passages Massive Text Embedding Benchmark