The Russian-focused embedders' exploration: ruMTEB benchmark and Russian
  embedding model design
Embedding models play a crucial role in Natural Language Processing (NLP) by
creating text embeddings used in various tasks such as information retrieval
and assessing semantic text similarity. This paper focuses on research related
to embedding models in the Russian language. It introduces a new
Russian-focused embedding model called ru-en-RoSBERTa and the ruMTEB benchmark,
the Russian version extending the Massive Text Embedding Benchmark (MTEB). Our
benchmark includes seven categories of tasks, such as semantic textual
similarity, text classification, reranking, and retrieval. The research also
assesses a representative set of Russian and multilingual models on the
proposed benchmark. The findings indicate that the new model achieves results
that are on par with state-of-the-art models in Russian. We release the model
ru-en-RoSBERTa, and the ruMTEB framework comes with open-source code,
integration into the original framework and a public leaderboard.
66c832db840b9278c5b8fa91
Artem Snegirev
extracted_pending
2024-08-23T06:57:32.477Z
5f8ed83d5d083370c711f642
/avatars/d0cf3e587a64feed7c7ea7870f0deb5a.svg
Artem Snegirev
user
artemsnegirev
66c832db840b9278c5b8fa92
Maria Tikhonova
66c832db840b9278c5b8fa93
Anna Maksimova
66c832db840b9278c5b8fa94
Alena Fenogenova
66c832db840b9278c5b8fa95
Alexander Abramov
claimed_verified
2024-08-26T07:15:12.748Z
639bf7c0445b133a4e98544a
https://cdn-avatars.huggingface.co/v1/production/uploads/1671165672918-noauth.jpeg
Aleksandr Abramov
user
Andrilko
The Russian-focused embedders' exploration: ruMTEB benchmark and Russian
  embedding model design
Embedding models play a crucial role in Natural Language Processing (NLP) by
creating text embeddings used in various tasks such as information retrieval
and assessing semantic text similarity. This paper focuses on research related
to embedding models in the Russian language. It introduces a new
Russian-focused embedding model called ru-en-RoSBERTa and the ruMTEB benchmark,
the Russian version extending the Massive Text Embedding Benchmark (MTEB). Our
benchmark includes seven categories of tasks, such as semantic textual
similarity, text classification, reranking, and retrieval. The research also
assesses a representative set of Russian and multilingual models on the
proposed benchmark. The findings indicate that the new model achieves results
that are on par with state-of-the-art models in Russian. We release the model
ru-en-RoSBERTa, and the ruMTEB framework comes with open-source code,
integration into the original framework and a public leaderboard.
A new Russian-focused embedding model, ru-en-RoSBERTa, and a Russian benchmark, ruMTEB, are introduced, demonstrating competitive performance in various NLP tasks compared to existing state-of-the-art models.
embedding models Natural Language Processing NLP ru-en-RoSBERTa ruMTEB Massive Text Embedding Benchmark MTEB semantic textual similarity text classification reranking retrieval