Your Mixture-of-Experts LLM Is Secretly an Embedding Model For Free
While large language models (LLMs) excel on generation tasks, their
decoder-only architecture often limits their potential as embedding models if
no further representation finetuning is applied. Does this contradict their
claim of generalists? To answer the question, we take a closer look at
Mixture-of-Experts (MoE) LLMs. Our study shows that the expert routers in MoE
LLMs can serve as an off-the-shelf embedding model with promising performance
on a diverse class of embedding-focused tasks, without requiring any
finetuning. Moreover, our extensive analysis shows that the MoE routing weights
(RW) is complementary to the hidden state (HS) of LLMs, a widely-used
embedding. Compared to HS, we find that RW is more robust to the choice of
prompts and focuses on high-level semantics. Motivated by the analysis, we
propose MoEE combining RW and HS, which achieves better performance than using
either separately. Our exploration of their combination and prompting strategy
shed several novel insights, e.g., a weighted sum of RW and HS similarities
outperforms the similarity on their concatenation. Our experiments are
conducted on 6 embedding tasks with 20 datasets from the Massive Text Embedding
Benchmark (MTEB). The results demonstrate the significant improvement brought
by MoEE to LLM-based embedding without further finetuning.
670e2ff1ffc411cfa85d2f1f
Ziyue Li
670e2ff1ffc411cfa85d2f20
Tianyi Zhou
claimed_verified
2024-10-16T09:44:17.073Z
647f5af5b0e96764589f3b2a
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/VJ4cDyjp5M3V5WmI5gPIU.jpeg
Tianyi Zhou
user
zhoutianyi
Your Mixture-of-Experts LLM Is Secretly an Embedding Model For Free
While large language models (LLMs) excel on generation tasks, their
decoder-only architecture often limits their potential as embedding models if
no further representation finetuning is applied. Does this contradict their
claim of generalists? To answer the question, we take a closer look at
Mixture-of-Experts (MoE) LLMs. Our study shows that the expert routers in MoE
LLMs can serve as an off-the-shelf embedding model with promising performance
on a diverse class of embedding-focused tasks, without requiring any
finetuning. Moreover, our extensive analysis shows that the MoE routing weights
(RW) is complementary to the hidden state (HS) of LLMs, a widely-used
embedding. Compared to HS, we find that RW is more robust to the choice of
prompts and focuses on high-level semantics. Motivated by the analysis, we
propose MoEE combining RW and HS, which achieves better performance than using
either separately. Our exploration of their combination and prompting strategy
shed several novel insights, e.g., a weighted sum of RW and HS similarities
outperforms the similarity on their concatenation. Our experiments are
conducted on 6 embedding tasks with 20 datasets from the Massive Text Embedding
Benchmark (MTEB). The results demonstrate the significant improvement brought
by MoEE to LLM-based embedding without further finetuning.
Mixture-of-Experts (MoE) LLMs can serve as effective embedding models without finetuning, with their routing weights and hidden states complementing each other to enhance performance.
Mixture-of-Experts (MoE) embedding models expert routers routing weights (RW) hidden state (HS) MoEE Massive Text Embedding Benchmark (MTEB)