Improving Text Embeddings for Smaller Language Models Using Contrastive
  Fine-tuning
While Large Language Models show remarkable performance in natural language
understanding, their resource-intensive nature makes them less accessible. In
contrast, smaller language models such as MiniCPM offer more sustainable
scalability, but often underperform without specialized optimization. In this
paper, we explore the enhancement of smaller language models through the
improvement of their text embeddings. We select three language models, MiniCPM,
Phi-2, and Gemma, to conduct contrastive fine-tuning on the NLI dataset. Our
results demonstrate that this fine-tuning method enhances the quality of text
embeddings for all three models across various benchmarks, with MiniCPM showing
the most significant improvements of an average 56.33\% performance gain. The
contrastive fine-tuning code is publicly available at
https://github.com/trapoom555/Language-Model-STS-CFT.
66ac43e305560e24ec2701e0
Trapoom Ukarapol
admin_assigned
2024-08-02T08:47:33.876Z
661cae1bd7c07238c295352b
https://cdn-avatars.huggingface.co/v1/production/uploads/661cae1bd7c07238c295352b/_ZYxfOMyrUlkxSjx0WBwG.jpeg
Trapoom Ukarapol
user
trapoom555
66ac43e305560e24ec2701e1
Zhicheng Lee
admin_assigned
2024-08-02T08:47:44.296Z
652542861e9db26e407aa1fc
/avatars/4c47ef4564f498a7f34b4a17a1e209a8.svg
Lee Zhicheng
user
ZhiCheng0326
66ac43e305560e24ec2701e2
Amy Xin
admin_assigned
2024-08-02T08:47:50.305Z
660bf98c3336a7e128a0e918
/avatars/3e3f2886bd4a730ec19b13aecc99279f.svg
Amy Xin
user
amyxx2001
Improving Text Embeddings for Smaller Language Models Using Contrastive
  Fine-tuning
While Large Language Models show remarkable performance in natural language
understanding, their resource-intensive nature makes them less accessible. In
contrast, smaller language models such as MiniCPM offer more sustainable
scalability, but often underperform without specialized optimization. In this
paper, we explore the enhancement of smaller language models through the
improvement of their text embeddings. We select three language models, MiniCPM,
Phi-2, and Gemma, to conduct contrastive fine-tuning on the NLI dataset. Our
results demonstrate that this fine-tuning method enhances the quality of text
embeddings for all three models across various benchmarks, with MiniCPM showing
the most significant improvements of an average 56.33\% performance gain. The
contrastive fine-tuning code is publicly available at
https://github.com/trapoom555/Language-Model-STS-CFT.
Contrastive fine-tuning improves text embeddings and performance in smaller language models across various benchmarks.
Large Language Models MiniCPM Phi-2 Gemma contrastive fine-tuning NLI dataset