HUME: Measuring the Human-Model Performance Gap in Text Embedding Task
Comparing human and model performance offers a valuable perspective for
understanding the strengths and limitations of embedding models, highlighting
where they succeed and where they fail to capture meaning and nuance. However,
such comparisons are rarely made, as human performance on embedding tasks is
difficult to measure. To fill this gap, we introduce HUME: Human Evaluation
Framework for Text Embeddings. While frameworks like MTEB provide broad model
evaluation, they lack reliable estimates of human performance, limiting the
interpretability of model scores. We measure human performance across 16 MTEB
datasets spanning reranking, classification, clustering, and semantic textual
similarity across linguistically diverse high- and low-resource languages.
Humans achieve an average performance of 77.6% compared to 80.1% for the best
embedding model, although variation is substantial: models reach near-ceiling
performance on some datasets while struggling on others, suggesting dataset
issues and revealing shortcomings in low-resource languages. We provide human
performance baselines, insight into task difficulty patterns, and an extensible
evaluation framework that enables a more meaningful interpretation of the model
and informs the development of both models and benchmarks. Our code, dataset,
and leaderboard are publicly available at
https://github.com/embeddings-benchmark/mteb.
68ede08ede1fee572713a9af
Adnan El Assadi
claimed_verified
2025-10-15T03:13:52.224Z
6671be9ff022d14aa10df864
/avatars/dd085abefa38c1604dc2ceabf472816d.svg
Adnan El Assadi
user
AdnanElAssadi
68ede08ede1fee572713a9b0
Isaac Chung
claimed_verified
2025-10-14T07:29:22.077Z
64cc0e80a257a3212c0c4b24
https://cdn-avatars.huggingface.co/v1/production/uploads/64cc0e80a257a3212c0c4b24/wqs6WZN8-3OQthcnQXgN7.png
Isaac Chung
user
isaacchung
68ede08ede1fee572713a9b1
Roman Solomatin
claimed_verified
2025-10-14T07:29:19.917Z
61af4544d691b3aadd1f62b6
/avatars/7a4067accdd1005f78c3c4adad3ee0a5.svg
Solomatin Roman
user
Samoed
68ede08ede1fee572713a9b2
Niklas Muennighoff
68ede08ede1fee572713a9b3
Kenneth Enevoldsen
claimed_verified
2025-10-14T10:23:39.431Z
5ff5943752c26e9bc240bada
https://cdn-avatars.huggingface.co/v1/production/uploads/5ff5943752c26e9bc240bada/Exyzf3C_gJ2KdsL4K5_cq.png
Kenneth C. Enevoldsen
user
KennethEnevoldsen
HUME: Measuring the Human-Model Performance Gap in Text Embedding Task
Comparing human and model performance offers a valuable perspective for
understanding the strengths and limitations of embedding models, highlighting
where they succeed and where they fail to capture meaning and nuance. However,
such comparisons are rarely made, as human performance on embedding tasks is
difficult to measure. To fill this gap, we introduce HUME: Human Evaluation
Framework for Text Embeddings. While frameworks like MTEB provide broad model
evaluation, they lack reliable estimates of human performance, limiting the
interpretability of model scores. We measure human performance across 16 MTEB
datasets spanning reranking, classification, clustering, and semantic textual
similarity across linguistically diverse high- and low-resource languages.
Humans achieve an average performance of 77.6% compared to 80.1% for the best
embedding model, although variation is substantial: models reach near-ceiling
performance on some datasets while struggling on others, suggesting dataset
issues and revealing shortcomings in low-resource languages. We provide human
performance baselines, insight into task difficulty patterns, and an extensible
evaluation framework that enables a more meaningful interpretation of the model
and informs the development of both models and benchmarks. Our code, dataset,
and leaderboard are publicly available at
https://github.com/embeddings-benchmark/mteb.
HUME provides human performance baselines for text embedding tasks, enhancing the interpretability of model evaluations and revealing dataset and language-specific challenges.
embedding models HUME Human Evaluation Framework for Text Embeddings MTEB reranking classification clustering semantic textual similarity low-resource languages dataset issues model development benchmarks