|
|
--- |
|
|
tags: |
|
|
- sentence-transformers |
|
|
- sentence-similarity |
|
|
- feature-extraction |
|
|
- generated_from_trainer |
|
|
- dataset_size:124788 |
|
|
- loss:GISTEmbedLoss |
|
|
base_model: BAAI/bge-small-en-v1.5 |
|
|
widget: |
|
|
- source_sentence: 其他机械、设备和有形货物租赁服务代表 |
|
|
sentences: |
|
|
- 其他机械和设备租赁服务工作人员 |
|
|
- 电子和电信设备及零部件物流经理 |
|
|
- 工业主厨 |
|
|
- source_sentence: 公交车司机 |
|
|
sentences: |
|
|
- 表演灯光设计师 |
|
|
- 乙烯基地板安装工 |
|
|
- 国际巴士司机 |
|
|
- source_sentence: online communication manager |
|
|
sentences: |
|
|
- trades union official |
|
|
- social media manager |
|
|
- budget manager |
|
|
- source_sentence: Projektmanagerin |
|
|
sentences: |
|
|
- Projektmanager/Projektmanagerin |
|
|
- Category-Manager |
|
|
- Infanterist |
|
|
- source_sentence: Volksvertreter |
|
|
sentences: |
|
|
- Parlamentarier |
|
|
- Oberbürgermeister |
|
|
- Konsul |
|
|
pipeline_tag: sentence-similarity |
|
|
library_name: sentence-transformers |
|
|
metrics: |
|
|
- cosine_accuracy@1 |
|
|
- cosine_accuracy@20 |
|
|
- cosine_accuracy@50 |
|
|
- cosine_accuracy@100 |
|
|
- cosine_accuracy@150 |
|
|
- cosine_accuracy@200 |
|
|
- cosine_precision@1 |
|
|
- cosine_precision@20 |
|
|
- cosine_precision@50 |
|
|
- cosine_precision@100 |
|
|
- cosine_precision@150 |
|
|
- cosine_precision@200 |
|
|
- cosine_recall@1 |
|
|
- cosine_recall@20 |
|
|
- cosine_recall@50 |
|
|
- cosine_recall@100 |
|
|
- cosine_recall@150 |
|
|
- cosine_recall@200 |
|
|
- cosine_ndcg@1 |
|
|
- cosine_ndcg@20 |
|
|
- cosine_ndcg@50 |
|
|
- cosine_ndcg@100 |
|
|
- cosine_ndcg@150 |
|
|
- cosine_ndcg@200 |
|
|
- cosine_mrr@1 |
|
|
- cosine_mrr@20 |
|
|
- cosine_mrr@50 |
|
|
- cosine_mrr@100 |
|
|
- cosine_mrr@150 |
|
|
- cosine_mrr@200 |
|
|
- cosine_map@1 |
|
|
- cosine_map@20 |
|
|
- cosine_map@50 |
|
|
- cosine_map@100 |
|
|
- cosine_map@150 |
|
|
- cosine_map@200 |
|
|
- cosine_map@500 |
|
|
model-index: |
|
|
- name: SentenceTransformer based on BAAI/bge-small-en-v1.5 |
|
|
results: |
|
|
- task: |
|
|
type: information-retrieval |
|
|
name: Information Retrieval |
|
|
dataset: |
|
|
name: full en |
|
|
type: full_en |
|
|
metrics: |
|
|
- type: cosine_accuracy@1 |
|
|
value: 0.6571428571428571 |
|
|
name: Cosine Accuracy@1 |
|
|
- type: cosine_accuracy@20 |
|
|
value: 0.9904761904761905 |
|
|
name: Cosine Accuracy@20 |
|
|
- type: cosine_accuracy@50 |
|
|
value: 0.9904761904761905 |
|
|
name: Cosine Accuracy@50 |
|
|
- type: cosine_accuracy@100 |
|
|
value: 0.9904761904761905 |
|
|
name: Cosine Accuracy@100 |
|
|
- type: cosine_accuracy@150 |
|
|
value: 0.9904761904761905 |
|
|
name: Cosine Accuracy@150 |
|
|
- type: cosine_accuracy@200 |
|
|
value: 0.9904761904761905 |
|
|
name: Cosine Accuracy@200 |
|
|
- type: cosine_precision@1 |
|
|
value: 0.6571428571428571 |
|
|
name: Cosine Precision@1 |
|
|
- type: cosine_precision@20 |
|
|
value: 0.5023809523809524 |
|
|
name: Cosine Precision@20 |
|
|
- type: cosine_precision@50 |
|
|
value: 0.30800000000000005 |
|
|
name: Cosine Precision@50 |
|
|
- type: cosine_precision@100 |
|
|
value: 0.18628571428571428 |
|
|
name: Cosine Precision@100 |
|
|
- type: cosine_precision@150 |
|
|
value: 0.1321904761904762 |
|
|
name: Cosine Precision@150 |
|
|
- type: cosine_precision@200 |
|
|
value: 0.10295238095238096 |
|
|
name: Cosine Precision@200 |
|
|
- type: cosine_recall@1 |
|
|
value: 0.0680237860830842 |
|
|
name: Cosine Recall@1 |
|
|
- type: cosine_recall@20 |
|
|
value: 0.5384852963395483 |
|
|
name: Cosine Recall@20 |
|
|
- type: cosine_recall@50 |
|
|
value: 0.7260449077992874 |
|
|
name: Cosine Recall@50 |
|
|
- type: cosine_recall@100 |
|
|
value: 0.8328530702930984 |
|
|
name: Cosine Recall@100 |
|
|
- type: cosine_recall@150 |
|
|
value: 0.8745262490032277 |
|
|
name: Cosine Recall@150 |
|
|
- type: cosine_recall@200 |
|
|
value: 0.9056960100263424 |
|
|
name: Cosine Recall@200 |
|
|
- type: cosine_ndcg@1 |
|
|
value: 0.6571428571428571 |
|
|
name: Cosine Ndcg@1 |
|
|
- type: cosine_ndcg@20 |
|
|
value: 0.6845256340390302 |
|
|
name: Cosine Ndcg@20 |
|
|
- type: cosine_ndcg@50 |
|
|
value: 0.7040452093638513 |
|
|
name: Cosine Ndcg@50 |
|
|
- type: cosine_ndcg@100 |
|
|
value: 0.758935932285001 |
|
|
name: Cosine Ndcg@100 |
|
|
- type: cosine_ndcg@150 |
|
|
value: 0.7774414598948007 |
|
|
name: Cosine Ndcg@150 |
|
|
- type: cosine_ndcg@200 |
|
|
value: 0.7892946240668293 |
|
|
name: Cosine Ndcg@200 |
|
|
- type: cosine_mrr@1 |
|
|
value: 0.6571428571428571 |
|
|
name: Cosine Mrr@1 |
|
|
- type: cosine_mrr@20 |
|
|
value: 0.8103174603174604 |
|
|
name: Cosine Mrr@20 |
|
|
- type: cosine_mrr@50 |
|
|
value: 0.8103174603174604 |
|
|
name: Cosine Mrr@50 |
|
|
- type: cosine_mrr@100 |
|
|
value: 0.8103174603174604 |
|
|
name: Cosine Mrr@100 |
|
|
- type: cosine_mrr@150 |
|
|
value: 0.8103174603174604 |
|
|
name: Cosine Mrr@150 |
|
|
- type: cosine_mrr@200 |
|
|
value: 0.8103174603174604 |
|
|
name: Cosine Mrr@200 |
|
|
- type: cosine_map@1 |
|
|
value: 0.6571428571428571 |
|
|
name: Cosine Map@1 |
|
|
- type: cosine_map@20 |
|
|
value: 0.5418235787800474 |
|
|
name: Cosine Map@20 |
|
|
- type: cosine_map@50 |
|
|
value: 0.5327215779103721 |
|
|
name: Cosine Map@50 |
|
|
- type: cosine_map@100 |
|
|
value: 0.565706253334091 |
|
|
name: Cosine Map@100 |
|
|
- type: cosine_map@150 |
|
|
value: 0.5733951147399983 |
|
|
name: Cosine Map@150 |
|
|
- type: cosine_map@200 |
|
|
value: 0.5771587776237981 |
|
|
name: Cosine Map@200 |
|
|
- type: cosine_map@500 |
|
|
value: 0.5813892452974444 |
|
|
name: Cosine Map@500 |
|
|
- task: |
|
|
type: information-retrieval |
|
|
name: Information Retrieval |
|
|
dataset: |
|
|
name: full es |
|
|
type: full_es |
|
|
metrics: |
|
|
- type: cosine_accuracy@1 |
|
|
value: 0.12432432432432433 |
|
|
name: Cosine Accuracy@1 |
|
|
- type: cosine_accuracy@20 |
|
|
value: 1.0 |
|
|
name: Cosine Accuracy@20 |
|
|
- type: cosine_accuracy@50 |
|
|
value: 1.0 |
|
|
name: Cosine Accuracy@50 |
|
|
- type: cosine_accuracy@100 |
|
|
value: 1.0 |
|
|
name: Cosine Accuracy@100 |
|
|
- type: cosine_accuracy@150 |
|
|
value: 1.0 |
|
|
name: Cosine Accuracy@150 |
|
|
- type: cosine_accuracy@200 |
|
|
value: 1.0 |
|
|
name: Cosine Accuracy@200 |
|
|
- type: cosine_precision@1 |
|
|
value: 0.12432432432432433 |
|
|
name: Cosine Precision@1 |
|
|
- type: cosine_precision@20 |
|
|
value: 0.4897297297297297 |
|
|
name: Cosine Precision@20 |
|
|
- type: cosine_precision@50 |
|
|
value: 0.31794594594594594 |
|
|
name: Cosine Precision@50 |
|
|
- type: cosine_precision@100 |
|
|
value: 0.19864864864864865 |
|
|
name: Cosine Precision@100 |
|
|
- type: cosine_precision@150 |
|
|
value: 0.14688288288288287 |
|
|
name: Cosine Precision@150 |
|
|
- type: cosine_precision@200 |
|
|
value: 0.11789189189189188 |
|
|
name: Cosine Precision@200 |
|
|
- type: cosine_recall@1 |
|
|
value: 0.003111544931768446 |
|
|
name: Cosine Recall@1 |
|
|
- type: cosine_recall@20 |
|
|
value: 0.32208664960961075 |
|
|
name: Cosine Recall@20 |
|
|
- type: cosine_recall@50 |
|
|
value: 0.46383117404893587 |
|
|
name: Cosine Recall@50 |
|
|
- type: cosine_recall@100 |
|
|
value: 0.5437537828683688 |
|
|
name: Cosine Recall@100 |
|
|
- type: cosine_recall@150 |
|
|
value: 0.5824968655076911 |
|
|
name: Cosine Recall@150 |
|
|
- type: cosine_recall@200 |
|
|
value: 0.6146962508233631 |
|
|
name: Cosine Recall@200 |
|
|
- type: cosine_ndcg@1 |
|
|
value: 0.12432432432432433 |
|
|
name: Cosine Ndcg@1 |
|
|
- type: cosine_ndcg@20 |
|
|
value: 0.5384577730264963 |
|
|
name: Cosine Ndcg@20 |
|
|
- type: cosine_ndcg@50 |
|
|
value: 0.5012455261232941 |
|
|
name: Cosine Ndcg@50 |
|
|
- type: cosine_ndcg@100 |
|
|
value: 0.5147486871284331 |
|
|
name: Cosine Ndcg@100 |
|
|
- type: cosine_ndcg@150 |
|
|
value: 0.5348194013794069 |
|
|
name: Cosine Ndcg@150 |
|
|
- type: cosine_ndcg@200 |
|
|
value: 0.5505397598095297 |
|
|
name: Cosine Ndcg@200 |
|
|
- type: cosine_mrr@1 |
|
|
value: 0.12432432432432433 |
|
|
name: Cosine Mrr@1 |
|
|
- type: cosine_mrr@20 |
|
|
value: 0.5515015015015016 |
|
|
name: Cosine Mrr@20 |
|
|
- type: cosine_mrr@50 |
|
|
value: 0.5515015015015016 |
|
|
name: Cosine Mrr@50 |
|
|
- type: cosine_mrr@100 |
|
|
value: 0.5515015015015016 |
|
|
name: Cosine Mrr@100 |
|
|
- type: cosine_mrr@150 |
|
|
value: 0.5515015015015016 |
|
|
name: Cosine Mrr@150 |
|
|
- type: cosine_mrr@200 |
|
|
value: 0.5515015015015016 |
|
|
name: Cosine Mrr@200 |
|
|
- type: cosine_map@1 |
|
|
value: 0.12432432432432433 |
|
|
name: Cosine Map@1 |
|
|
- type: cosine_map@20 |
|
|
value: 0.40280623036556984 |
|
|
name: Cosine Map@20 |
|
|
- type: cosine_map@50 |
|
|
value: 0.3421710529569103 |
|
|
name: Cosine Map@50 |
|
|
- type: cosine_map@100 |
|
|
value: 0.33947884152876345 |
|
|
name: Cosine Map@100 |
|
|
- type: cosine_map@150 |
|
|
value: 0.34777364049184706 |
|
|
name: Cosine Map@150 |
|
|
- type: cosine_map@200 |
|
|
value: 0.35339765423089375 |
|
|
name: Cosine Map@200 |
|
|
- type: cosine_map@500 |
|
|
value: 0.3631043007370563 |
|
|
name: Cosine Map@500 |
|
|
- task: |
|
|
type: information-retrieval |
|
|
name: Information Retrieval |
|
|
dataset: |
|
|
name: full de |
|
|
type: full_de |
|
|
metrics: |
|
|
- type: cosine_accuracy@1 |
|
|
value: 0.2955665024630542 |
|
|
name: Cosine Accuracy@1 |
|
|
- type: cosine_accuracy@20 |
|
|
value: 0.9211822660098522 |
|
|
name: Cosine Accuracy@20 |
|
|
- type: cosine_accuracy@50 |
|
|
value: 0.9655172413793104 |
|
|
name: Cosine Accuracy@50 |
|
|
- type: cosine_accuracy@100 |
|
|
value: 0.9753694581280788 |
|
|
name: Cosine Accuracy@100 |
|
|
- type: cosine_accuracy@150 |
|
|
value: 0.9852216748768473 |
|
|
name: Cosine Accuracy@150 |
|
|
- type: cosine_accuracy@200 |
|
|
value: 0.9852216748768473 |
|
|
name: Cosine Accuracy@200 |
|
|
- type: cosine_precision@1 |
|
|
value: 0.2955665024630542 |
|
|
name: Cosine Precision@1 |
|
|
- type: cosine_precision@20 |
|
|
value: 0.4246305418719211 |
|
|
name: Cosine Precision@20 |
|
|
- type: cosine_precision@50 |
|
|
value: 0.2813793103448276 |
|
|
name: Cosine Precision@50 |
|
|
- type: cosine_precision@100 |
|
|
value: 0.1800985221674877 |
|
|
name: Cosine Precision@100 |
|
|
- type: cosine_precision@150 |
|
|
value: 0.1362233169129721 |
|
|
name: Cosine Precision@150 |
|
|
- type: cosine_precision@200 |
|
|
value: 0.11054187192118226 |
|
|
name: Cosine Precision@200 |
|
|
- type: cosine_recall@1 |
|
|
value: 0.01108543831680986 |
|
|
name: Cosine Recall@1 |
|
|
- type: cosine_recall@20 |
|
|
value: 0.26139377973111655 |
|
|
name: Cosine Recall@20 |
|
|
- type: cosine_recall@50 |
|
|
value: 0.3835171819041212 |
|
|
name: Cosine Recall@50 |
|
|
- type: cosine_recall@100 |
|
|
value: 0.4676892706124872 |
|
|
name: Cosine Recall@100 |
|
|
- type: cosine_recall@150 |
|
|
value: 0.5183014504752351 |
|
|
name: Cosine Recall@150 |
|
|
- type: cosine_recall@200 |
|
|
value: 0.551717511250073 |
|
|
name: Cosine Recall@200 |
|
|
- type: cosine_ndcg@1 |
|
|
value: 0.2955665024630542 |
|
|
name: Cosine Ndcg@1 |
|
|
- type: cosine_ndcg@20 |
|
|
value: 0.4600580109269636 |
|
|
name: Cosine Ndcg@20 |
|
|
- type: cosine_ndcg@50 |
|
|
value: 0.4229190542750304 |
|
|
name: Cosine Ndcg@50 |
|
|
- type: cosine_ndcg@100 |
|
|
value: 0.4370543021366767 |
|
|
name: Cosine Ndcg@100 |
|
|
- type: cosine_ndcg@150 |
|
|
value: 0.46289045418097646 |
|
|
name: Cosine Ndcg@150 |
|
|
- type: cosine_ndcg@200 |
|
|
value: 0.4796711024513544 |
|
|
name: Cosine Ndcg@200 |
|
|
- type: cosine_mrr@1 |
|
|
value: 0.2955665024630542 |
|
|
name: Cosine Mrr@1 |
|
|
- type: cosine_mrr@20 |
|
|
value: 0.48958320005117995 |
|
|
name: Cosine Mrr@20 |
|
|
- type: cosine_mrr@50 |
|
|
value: 0.49093477998292195 |
|
|
name: Cosine Mrr@50 |
|
|
- type: cosine_mrr@100 |
|
|
value: 0.4910841931964832 |
|
|
name: Cosine Mrr@100 |
|
|
- type: cosine_mrr@150 |
|
|
value: 0.4911623560854821 |
|
|
name: Cosine Mrr@150 |
|
|
- type: cosine_mrr@200 |
|
|
value: 0.4911623560854821 |
|
|
name: Cosine Mrr@200 |
|
|
- type: cosine_map@1 |
|
|
value: 0.2955665024630542 |
|
|
name: Cosine Map@1 |
|
|
- type: cosine_map@20 |
|
|
value: 0.32364842421740225 |
|
|
name: Cosine Map@20 |
|
|
- type: cosine_map@50 |
|
|
value: 0.2643813390551392 |
|
|
name: Cosine Map@50 |
|
|
- type: cosine_map@100 |
|
|
value: 0.2576413544507463 |
|
|
name: Cosine Map@100 |
|
|
- type: cosine_map@150 |
|
|
value: 0.2669126239698539 |
|
|
name: Cosine Map@150 |
|
|
- type: cosine_map@200 |
|
|
value: 0.27215799504041416 |
|
|
name: Cosine Map@200 |
|
|
- type: cosine_map@500 |
|
|
value: 0.28329484592874316 |
|
|
name: Cosine Map@500 |
|
|
- task: |
|
|
type: information-retrieval |
|
|
name: Information Retrieval |
|
|
dataset: |
|
|
name: full zh |
|
|
type: full_zh |
|
|
metrics: |
|
|
- type: cosine_accuracy@1 |
|
|
value: 0.34951456310679613 |
|
|
name: Cosine Accuracy@1 |
|
|
- type: cosine_accuracy@20 |
|
|
value: 0.7378640776699029 |
|
|
name: Cosine Accuracy@20 |
|
|
- type: cosine_accuracy@50 |
|
|
value: 0.8252427184466019 |
|
|
name: Cosine Accuracy@50 |
|
|
- type: cosine_accuracy@100 |
|
|
value: 0.8543689320388349 |
|
|
name: Cosine Accuracy@100 |
|
|
- type: cosine_accuracy@150 |
|
|
value: 0.9029126213592233 |
|
|
name: Cosine Accuracy@150 |
|
|
- type: cosine_accuracy@200 |
|
|
value: 0.941747572815534 |
|
|
name: Cosine Accuracy@200 |
|
|
- type: cosine_precision@1 |
|
|
value: 0.34951456310679613 |
|
|
name: Cosine Precision@1 |
|
|
- type: cosine_precision@20 |
|
|
value: 0.17330097087378643 |
|
|
name: Cosine Precision@20 |
|
|
- type: cosine_precision@50 |
|
|
value: 0.09436893203883494 |
|
|
name: Cosine Precision@50 |
|
|
- type: cosine_precision@100 |
|
|
value: 0.05893203883495146 |
|
|
name: Cosine Precision@100 |
|
|
- type: cosine_precision@150 |
|
|
value: 0.0458252427184466 |
|
|
name: Cosine Precision@150 |
|
|
- type: cosine_precision@200 |
|
|
value: 0.03854368932038834 |
|
|
name: Cosine Precision@200 |
|
|
- type: cosine_recall@1 |
|
|
value: 0.02726635297033844 |
|
|
name: Cosine Recall@1 |
|
|
- type: cosine_recall@20 |
|
|
value: 0.17661061398990294 |
|
|
name: Cosine Recall@20 |
|
|
- type: cosine_recall@50 |
|
|
value: 0.2392861843604663 |
|
|
name: Cosine Recall@50 |
|
|
- type: cosine_recall@100 |
|
|
value: 0.2862639658547104 |
|
|
name: Cosine Recall@100 |
|
|
- type: cosine_recall@150 |
|
|
value: 0.3286954340443375 |
|
|
name: Cosine Recall@150 |
|
|
- type: cosine_recall@200 |
|
|
value: 0.3630829587412431 |
|
|
name: Cosine Recall@200 |
|
|
- type: cosine_ndcg@1 |
|
|
value: 0.34951456310679613 |
|
|
name: Cosine Ndcg@1 |
|
|
- type: cosine_ndcg@20 |
|
|
value: 0.24683538489164747 |
|
|
name: Cosine Ndcg@20 |
|
|
- type: cosine_ndcg@50 |
|
|
value: 0.23936442282824424 |
|
|
name: Cosine Ndcg@50 |
|
|
- type: cosine_ndcg@100 |
|
|
value: 0.2618891246293786 |
|
|
name: Cosine Ndcg@100 |
|
|
- type: cosine_ndcg@150 |
|
|
value: 0.27867525817923894 |
|
|
name: Cosine Ndcg@150 |
|
|
- type: cosine_ndcg@200 |
|
|
value: 0.29190260238165355 |
|
|
name: Cosine Ndcg@200 |
|
|
- type: cosine_mrr@1 |
|
|
value: 0.34951456310679613 |
|
|
name: Cosine Mrr@1 |
|
|
- type: cosine_mrr@20 |
|
|
value: 0.44845699819699636 |
|
|
name: Cosine Mrr@20 |
|
|
- type: cosine_mrr@50 |
|
|
value: 0.4514515915598798 |
|
|
name: Cosine Mrr@50 |
|
|
- type: cosine_mrr@100 |
|
|
value: 0.451864194979824 |
|
|
name: Cosine Mrr@100 |
|
|
- type: cosine_mrr@150 |
|
|
value: 0.4522894025156287 |
|
|
name: Cosine Mrr@150 |
|
|
- type: cosine_mrr@200 |
|
|
value: 0.45250948321580986 |
|
|
name: Cosine Mrr@200 |
|
|
- type: cosine_map@1 |
|
|
value: 0.34951456310679613 |
|
|
name: Cosine Map@1 |
|
|
- type: cosine_map@20 |
|
|
value: 0.1470309927546457 |
|
|
name: Cosine Map@20 |
|
|
- type: cosine_map@50 |
|
|
value: 0.12671489844037503 |
|
|
name: Cosine Map@50 |
|
|
- type: cosine_map@100 |
|
|
value: 0.13257859039926595 |
|
|
name: Cosine Map@100 |
|
|
- type: cosine_map@150 |
|
|
value: 0.13523273342027425 |
|
|
name: Cosine Map@150 |
|
|
- type: cosine_map@200 |
|
|
value: 0.13679857663871084 |
|
|
name: Cosine Map@200 |
|
|
- type: cosine_map@500 |
|
|
value: 0.14069476480399515 |
|
|
name: Cosine Map@500 |
|
|
- task: |
|
|
type: information-retrieval |
|
|
name: Information Retrieval |
|
|
dataset: |
|
|
name: mix es |
|
|
type: mix_es |
|
|
metrics: |
|
|
- type: cosine_accuracy@1 |
|
|
value: 0.41133645345813835 |
|
|
name: Cosine Accuracy@1 |
|
|
- type: cosine_accuracy@20 |
|
|
value: 0.7613104524180967 |
|
|
name: Cosine Accuracy@20 |
|
|
- type: cosine_accuracy@50 |
|
|
value: 0.8523140925637025 |
|
|
name: Cosine Accuracy@50 |
|
|
- type: cosine_accuracy@100 |
|
|
value: 0.9121164846593863 |
|
|
name: Cosine Accuracy@100 |
|
|
- type: cosine_accuracy@150 |
|
|
value: 0.9417576703068122 |
|
|
name: Cosine Accuracy@150 |
|
|
- type: cosine_accuracy@200 |
|
|
value: 0.9547581903276131 |
|
|
name: Cosine Accuracy@200 |
|
|
- type: cosine_precision@1 |
|
|
value: 0.41133645345813835 |
|
|
name: Cosine Precision@1 |
|
|
- type: cosine_precision@20 |
|
|
value: 0.08920956838273532 |
|
|
name: Cosine Precision@20 |
|
|
- type: cosine_precision@50 |
|
|
value: 0.04175767030681228 |
|
|
name: Cosine Precision@50 |
|
|
- type: cosine_precision@100 |
|
|
value: 0.02291731669266771 |
|
|
name: Cosine Precision@100 |
|
|
- type: cosine_precision@150 |
|
|
value: 0.015905702894782457 |
|
|
name: Cosine Precision@150 |
|
|
- type: cosine_precision@200 |
|
|
value: 0.012243889755590227 |
|
|
name: Cosine Precision@200 |
|
|
- type: cosine_recall@1 |
|
|
value: 0.15653988064284477 |
|
|
name: Cosine Recall@1 |
|
|
- type: cosine_recall@20 |
|
|
value: 0.6593678032835598 |
|
|
name: Cosine Recall@20 |
|
|
- type: cosine_recall@50 |
|
|
value: 0.7704838669737266 |
|
|
name: Cosine Recall@50 |
|
|
- type: cosine_recall@100 |
|
|
value: 0.847169601069757 |
|
|
name: Cosine Recall@100 |
|
|
- type: cosine_recall@150 |
|
|
value: 0.8825483495530297 |
|
|
name: Cosine Recall@150 |
|
|
- type: cosine_recall@200 |
|
|
value: 0.9050999182824455 |
|
|
name: Cosine Recall@200 |
|
|
- type: cosine_ndcg@1 |
|
|
value: 0.41133645345813835 |
|
|
name: Cosine Ndcg@1 |
|
|
- type: cosine_ndcg@20 |
|
|
value: 0.5116672519515115 |
|
|
name: Cosine Ndcg@20 |
|
|
- type: cosine_ndcg@50 |
|
|
value: 0.542000920569141 |
|
|
name: Cosine Ndcg@50 |
|
|
- type: cosine_ndcg@100 |
|
|
value: 0.558759964344595 |
|
|
name: Cosine Ndcg@100 |
|
|
- type: cosine_ndcg@150 |
|
|
value: 0.5655977162199296 |
|
|
name: Cosine Ndcg@150 |
|
|
- type: cosine_ndcg@200 |
|
|
value: 0.5697289878952349 |
|
|
name: Cosine Ndcg@200 |
|
|
- type: cosine_mrr@1 |
|
|
value: 0.41133645345813835 |
|
|
name: Cosine Mrr@1 |
|
|
- type: cosine_mrr@20 |
|
|
value: 0.4978677179556957 |
|
|
name: Cosine Mrr@20 |
|
|
- type: cosine_mrr@50 |
|
|
value: 0.5009543893008301 |
|
|
name: Cosine Mrr@50 |
|
|
- type: cosine_mrr@100 |
|
|
value: 0.5018183607581652 |
|
|
name: Cosine Mrr@100 |
|
|
- type: cosine_mrr@150 |
|
|
value: 0.5020589846475842 |
|
|
name: Cosine Mrr@150 |
|
|
- type: cosine_mrr@200 |
|
|
value: 0.5021321446410069 |
|
|
name: Cosine Mrr@200 |
|
|
- type: cosine_map@1 |
|
|
value: 0.41133645345813835 |
|
|
name: Cosine Map@1 |
|
|
- type: cosine_map@20 |
|
|
value: 0.4263681424556441 |
|
|
name: Cosine Map@20 |
|
|
- type: cosine_map@50 |
|
|
value: 0.4338209025376249 |
|
|
name: Cosine Map@50 |
|
|
- type: cosine_map@100 |
|
|
value: 0.4359939776007631 |
|
|
name: Cosine Map@100 |
|
|
- type: cosine_map@150 |
|
|
value: 0.43656970643226983 |
|
|
name: Cosine Map@150 |
|
|
- type: cosine_map@200 |
|
|
value: 0.4368426702726571 |
|
|
name: Cosine Map@200 |
|
|
- type: cosine_map@500 |
|
|
value: 0.43729529920887905 |
|
|
name: Cosine Map@500 |
|
|
- task: |
|
|
type: information-retrieval |
|
|
name: Information Retrieval |
|
|
dataset: |
|
|
name: mix de |
|
|
type: mix_de |
|
|
metrics: |
|
|
- type: cosine_accuracy@1 |
|
|
value: 0.29433177327093085 |
|
|
name: Cosine Accuracy@1 |
|
|
- type: cosine_accuracy@20 |
|
|
value: 0.6500260010400416 |
|
|
name: Cosine Accuracy@20 |
|
|
- type: cosine_accuracy@50 |
|
|
value: 0.7607904316172647 |
|
|
name: Cosine Accuracy@50 |
|
|
- type: cosine_accuracy@100 |
|
|
value: 0.8507540301612064 |
|
|
name: Cosine Accuracy@100 |
|
|
- type: cosine_accuracy@150 |
|
|
value: 0.889755590223609 |
|
|
name: Cosine Accuracy@150 |
|
|
- type: cosine_accuracy@200 |
|
|
value: 0.9204368174726989 |
|
|
name: Cosine Accuracy@200 |
|
|
- type: cosine_precision@1 |
|
|
value: 0.29433177327093085 |
|
|
name: Cosine Precision@1 |
|
|
- type: cosine_precision@20 |
|
|
value: 0.07308892355694228 |
|
|
name: Cosine Precision@20 |
|
|
- type: cosine_precision@50 |
|
|
value: 0.036141445657826315 |
|
|
name: Cosine Precision@50 |
|
|
- type: cosine_precision@100 |
|
|
value: 0.020634425377015084 |
|
|
name: Cosine Precision@100 |
|
|
- type: cosine_precision@150 |
|
|
value: 0.014681920610157736 |
|
|
name: Cosine Precision@150 |
|
|
- type: cosine_precision@200 |
|
|
value: 0.011552262090483621 |
|
|
name: Cosine Precision@200 |
|
|
- type: cosine_recall@1 |
|
|
value: 0.1109031027907783 |
|
|
name: Cosine Recall@1 |
|
|
- type: cosine_recall@20 |
|
|
value: 0.534356040908303 |
|
|
name: Cosine Recall@20 |
|
|
- type: cosine_recall@50 |
|
|
value: 0.6584676720402148 |
|
|
name: Cosine Recall@50 |
|
|
- type: cosine_recall@100 |
|
|
value: 0.752470098803952 |
|
|
name: Cosine Recall@100 |
|
|
- type: cosine_recall@150 |
|
|
value: 0.8025567689374241 |
|
|
name: Cosine Recall@150 |
|
|
- type: cosine_recall@200 |
|
|
value: 0.8417663373201595 |
|
|
name: Cosine Recall@200 |
|
|
- type: cosine_ndcg@1 |
|
|
value: 0.29433177327093085 |
|
|
name: Cosine Ndcg@1 |
|
|
- type: cosine_ndcg@20 |
|
|
value: 0.3919428679123834 |
|
|
name: Cosine Ndcg@20 |
|
|
- type: cosine_ndcg@50 |
|
|
value: 0.425599899100406 |
|
|
name: Cosine Ndcg@50 |
|
|
- type: cosine_ndcg@100 |
|
|
value: 0.4462421162922913 |
|
|
name: Cosine Ndcg@100 |
|
|
- type: cosine_ndcg@150 |
|
|
value: 0.45606402272845137 |
|
|
name: Cosine Ndcg@150 |
|
|
- type: cosine_ndcg@200 |
|
|
value: 0.4632312746623382 |
|
|
name: Cosine Ndcg@200 |
|
|
- type: cosine_mrr@1 |
|
|
value: 0.29433177327093085 |
|
|
name: Cosine Mrr@1 |
|
|
- type: cosine_mrr@20 |
|
|
value: 0.37785395494554963 |
|
|
name: Cosine Mrr@20 |
|
|
- type: cosine_mrr@50 |
|
|
value: 0.38148321196953044 |
|
|
name: Cosine Mrr@50 |
|
|
- type: cosine_mrr@100 |
|
|
value: 0.38274724688611994 |
|
|
name: Cosine Mrr@100 |
|
|
- type: cosine_mrr@150 |
|
|
value: 0.3830666241433367 |
|
|
name: Cosine Mrr@150 |
|
|
- type: cosine_mrr@200 |
|
|
value: 0.3832429794087988 |
|
|
name: Cosine Mrr@200 |
|
|
- type: cosine_map@1 |
|
|
value: 0.29433177327093085 |
|
|
name: Cosine Map@1 |
|
|
- type: cosine_map@20 |
|
|
value: 0.3096720133634083 |
|
|
name: Cosine Map@20 |
|
|
- type: cosine_map@50 |
|
|
value: 0.31740714963039135 |
|
|
name: Cosine Map@50 |
|
|
- type: cosine_map@100 |
|
|
value: 0.31992557448195186 |
|
|
name: Cosine Map@100 |
|
|
- type: cosine_map@150 |
|
|
value: 0.3207379270967634 |
|
|
name: Cosine Map@150 |
|
|
- type: cosine_map@200 |
|
|
value: 0.3211962807999124 |
|
|
name: Cosine Map@200 |
|
|
- type: cosine_map@500 |
|
|
value: 0.3219246841517722 |
|
|
name: Cosine Map@500 |
|
|
- task: |
|
|
type: information-retrieval |
|
|
name: Information Retrieval |
|
|
dataset: |
|
|
name: mix zh |
|
|
type: mix_zh |
|
|
metrics: |
|
|
- type: cosine_accuracy@1 |
|
|
value: 0.09707724425887265 |
|
|
name: Cosine Accuracy@1 |
|
|
- type: cosine_accuracy@20 |
|
|
value: 0.3585594989561587 |
|
|
name: Cosine Accuracy@20 |
|
|
- type: cosine_accuracy@50 |
|
|
value: 0.4900835073068894 |
|
|
name: Cosine Accuracy@50 |
|
|
- type: cosine_accuracy@100 |
|
|
value: 0.6002087682672234 |
|
|
name: Cosine Accuracy@100 |
|
|
- type: cosine_accuracy@150 |
|
|
value: 0.6612734864300627 |
|
|
name: Cosine Accuracy@150 |
|
|
- type: cosine_accuracy@200 |
|
|
value: 0.7061586638830898 |
|
|
name: Cosine Accuracy@200 |
|
|
- type: cosine_precision@1 |
|
|
value: 0.09707724425887265 |
|
|
name: Cosine Precision@1 |
|
|
- type: cosine_precision@20 |
|
|
value: 0.03144572025052192 |
|
|
name: Cosine Precision@20 |
|
|
- type: cosine_precision@50 |
|
|
value: 0.018486430062630482 |
|
|
name: Cosine Precision@50 |
|
|
- type: cosine_precision@100 |
|
|
value: 0.011612734864300627 |
|
|
name: Cosine Precision@100 |
|
|
- type: cosine_precision@150 |
|
|
value: 0.008688239387613084 |
|
|
name: Cosine Precision@150 |
|
|
- type: cosine_precision@200 |
|
|
value: 0.007132045929018789 |
|
|
name: Cosine Precision@200 |
|
|
- type: cosine_recall@1 |
|
|
value: 0.032868575405109846 |
|
|
name: Cosine Recall@1 |
|
|
- type: cosine_recall@20 |
|
|
value: 0.20912118500845014 |
|
|
name: Cosine Recall@20 |
|
|
- type: cosine_recall@50 |
|
|
value: 0.305353414852371 |
|
|
name: Cosine Recall@50 |
|
|
- type: cosine_recall@100 |
|
|
value: 0.3834696126188819 |
|
|
name: Cosine Recall@100 |
|
|
- type: cosine_recall@150 |
|
|
value: 0.43087740663419155 |
|
|
name: Cosine Recall@150 |
|
|
- type: cosine_recall@200 |
|
|
value: 0.4714567385757365 |
|
|
name: Cosine Recall@200 |
|
|
- type: cosine_ndcg@1 |
|
|
value: 0.09707724425887265 |
|
|
name: Cosine Ndcg@1 |
|
|
- type: cosine_ndcg@20 |
|
|
value: 0.13847583254619214 |
|
|
name: Cosine Ndcg@20 |
|
|
- type: cosine_ndcg@50 |
|
|
value: 0.16556220177827802 |
|
|
name: Cosine Ndcg@50 |
|
|
- type: cosine_ndcg@100 |
|
|
value: 0.1834871578549362 |
|
|
name: Cosine Ndcg@100 |
|
|
- type: cosine_ndcg@150 |
|
|
value: 0.1930615498205831 |
|
|
name: Cosine Ndcg@150 |
|
|
- type: cosine_ndcg@200 |
|
|
value: 0.20074882110420836 |
|
|
name: Cosine Ndcg@200 |
|
|
- type: cosine_mrr@1 |
|
|
value: 0.09707724425887265 |
|
|
name: Cosine Mrr@1 |
|
|
- type: cosine_mrr@20 |
|
|
value: 0.15220960831749397 |
|
|
name: Cosine Mrr@20 |
|
|
- type: cosine_mrr@50 |
|
|
value: 0.15642354470896513 |
|
|
name: Cosine Mrr@50 |
|
|
- type: cosine_mrr@100 |
|
|
value: 0.1580041495008456 |
|
|
name: Cosine Mrr@100 |
|
|
- type: cosine_mrr@150 |
|
|
value: 0.15850022553236756 |
|
|
name: Cosine Mrr@150 |
|
|
- type: cosine_mrr@200 |
|
|
value: 0.1587557913720219 |
|
|
name: Cosine Mrr@200 |
|
|
- type: cosine_map@1 |
|
|
value: 0.09707724425887265 |
|
|
name: Cosine Map@1 |
|
|
- type: cosine_map@20 |
|
|
value: 0.08751052569766739 |
|
|
name: Cosine Map@20 |
|
|
- type: cosine_map@50 |
|
|
value: 0.09304075210745723 |
|
|
name: Cosine Map@50 |
|
|
- type: cosine_map@100 |
|
|
value: 0.09500635866296525 |
|
|
name: Cosine Map@100 |
|
|
- type: cosine_map@150 |
|
|
value: 0.09570276054684158 |
|
|
name: Cosine Map@150 |
|
|
- type: cosine_map@200 |
|
|
value: 0.09614394028730197 |
|
|
name: Cosine Map@200 |
|
|
- type: cosine_map@500 |
|
|
value: 0.09706713378133278 |
|
|
name: Cosine Map@500 |
|
|
--- |
|
|
|
|
|
# Job - Job matching BAAI/bge-small-en-v1.5 |
|
|
|
|
|
Top performing model on [TalentCLEF 2025](https://talentclef.github.io/talentclef/) Task A. Use it for multilingual job title matching |
|
|
|
|
|
## Model Details |
|
|
|
|
|
### Model Description |
|
|
- **Model Type:** Sentence Transformer |
|
|
- **Base model:** [BAAI/bge-small-en-v1.5](https://huggingface.co/BAAI/bge-small-en-v1.5) <!-- at revision 5c38ec7c405ec4b44b94cc5a9bb96e735b38267a --> |
|
|
- **Maximum Sequence Length:** 512 tokens |
|
|
- **Output Dimensionality:** 384 dimensions |
|
|
- **Similarity Function:** Cosine Similarity |
|
|
- **Training Datasets:** |
|
|
- full_en |
|
|
- full_de |
|
|
- full_es |
|
|
- full_zh |
|
|
- mix |
|
|
<!-- - **Language:** Unknown --> |
|
|
<!-- - **License:** Unknown --> |
|
|
|
|
|
### Model Sources |
|
|
|
|
|
- **Documentation:** [Sentence Transformers Documentation](https://sbert.net) |
|
|
- **Repository:** [Sentence Transformers on GitHub](https://github.com/UKPLab/sentence-transformers) |
|
|
- **Hugging Face:** [Sentence Transformers on Hugging Face](https://huggingface.co/models?library=sentence-transformers) |
|
|
|
|
|
### Full Model Architecture |
|
|
|
|
|
``` |
|
|
SentenceTransformer( |
|
|
(0): Transformer({'max_seq_length': 512, 'do_lower_case': True}) with Transformer model: BertModel |
|
|
(1): Pooling({'word_embedding_dimension': 384, 'pooling_mode_cls_token': True, 'pooling_mode_mean_tokens': False, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True}) |
|
|
(2): Normalize() |
|
|
) |
|
|
``` |
|
|
|
|
|
## Usage |
|
|
|
|
|
### Direct Usage (Sentence Transformers) |
|
|
|
|
|
First install the Sentence Transformers library: |
|
|
|
|
|
```bash |
|
|
pip install -U sentence-transformers |
|
|
``` |
|
|
|
|
|
Then you can load this model and run inference. |
|
|
```python |
|
|
from sentence_transformers import SentenceTransformer |
|
|
|
|
|
# Download from the 🤗 Hub |
|
|
model = SentenceTransformer("sentence_transformers_model_id") |
|
|
# Run inference |
|
|
sentences = [ |
|
|
'Volksvertreter', |
|
|
'Parlamentarier', |
|
|
'Oberbürgermeister', |
|
|
] |
|
|
embeddings = model.encode(sentences) |
|
|
print(embeddings.shape) |
|
|
# [3, 384] |
|
|
|
|
|
# Get the similarity scores for the embeddings |
|
|
similarities = model.similarity(embeddings, embeddings) |
|
|
print(similarities.shape) |
|
|
# [3, 3] |
|
|
``` |
|
|
|
|
|
<!-- |
|
|
### Direct Usage (Transformers) |
|
|
|
|
|
<details><summary>Click to see the direct usage in Transformers</summary> |
|
|
|
|
|
</details> |
|
|
--> |
|
|
|
|
|
<!-- |
|
|
### Downstream Usage (Sentence Transformers) |
|
|
|
|
|
You can finetune this model on your own dataset. |
|
|
|
|
|
<details><summary>Click to expand</summary> |
|
|
|
|
|
</details> |
|
|
--> |
|
|
|
|
|
<!-- |
|
|
### Out-of-Scope Use |
|
|
|
|
|
*List how the model may foreseeably be misused and address what users ought not to do with the model.* |
|
|
--> |
|
|
|
|
|
## Evaluation |
|
|
|
|
|
### Metrics |
|
|
|
|
|
#### Information Retrieval |
|
|
|
|
|
* Datasets: `full_en`, `full_es`, `full_de`, `full_zh`, `mix_es`, `mix_de` and `mix_zh` |
|
|
* Evaluated with [<code>InformationRetrievalEvaluator</code>](https://sbert.net/docs/package_reference/sentence_transformer/evaluation.html#sentence_transformers.evaluation.InformationRetrievalEvaluator) |
|
|
|
|
|
| Metric | full_en | full_es | full_de | full_zh | mix_es | mix_de | mix_zh | |
|
|
|:---------------------|:-----------|:-----------|:-----------|:-----------|:-----------|:-----------|:-----------| |
|
|
| cosine_accuracy@1 | 0.6571 | 0.1243 | 0.2956 | 0.3495 | 0.4113 | 0.2943 | 0.0971 | |
|
|
| cosine_accuracy@20 | 0.9905 | 1.0 | 0.9212 | 0.7379 | 0.7613 | 0.65 | 0.3586 | |
|
|
| cosine_accuracy@50 | 0.9905 | 1.0 | 0.9655 | 0.8252 | 0.8523 | 0.7608 | 0.4901 | |
|
|
| cosine_accuracy@100 | 0.9905 | 1.0 | 0.9754 | 0.8544 | 0.9121 | 0.8508 | 0.6002 | |
|
|
| cosine_accuracy@150 | 0.9905 | 1.0 | 0.9852 | 0.9029 | 0.9418 | 0.8898 | 0.6613 | |
|
|
| cosine_accuracy@200 | 0.9905 | 1.0 | 0.9852 | 0.9417 | 0.9548 | 0.9204 | 0.7062 | |
|
|
| cosine_precision@1 | 0.6571 | 0.1243 | 0.2956 | 0.3495 | 0.4113 | 0.2943 | 0.0971 | |
|
|
| cosine_precision@20 | 0.5024 | 0.4897 | 0.4246 | 0.1733 | 0.0892 | 0.0731 | 0.0314 | |
|
|
| cosine_precision@50 | 0.308 | 0.3179 | 0.2814 | 0.0944 | 0.0418 | 0.0361 | 0.0185 | |
|
|
| cosine_precision@100 | 0.1863 | 0.1986 | 0.1801 | 0.0589 | 0.0229 | 0.0206 | 0.0116 | |
|
|
| cosine_precision@150 | 0.1322 | 0.1469 | 0.1362 | 0.0458 | 0.0159 | 0.0147 | 0.0087 | |
|
|
| cosine_precision@200 | 0.103 | 0.1179 | 0.1105 | 0.0385 | 0.0122 | 0.0116 | 0.0071 | |
|
|
| cosine_recall@1 | 0.068 | 0.0031 | 0.0111 | 0.0273 | 0.1565 | 0.1109 | 0.0329 | |
|
|
| cosine_recall@20 | 0.5385 | 0.3221 | 0.2614 | 0.1766 | 0.6594 | 0.5344 | 0.2091 | |
|
|
| cosine_recall@50 | 0.726 | 0.4638 | 0.3835 | 0.2393 | 0.7705 | 0.6585 | 0.3054 | |
|
|
| cosine_recall@100 | 0.8329 | 0.5438 | 0.4677 | 0.2863 | 0.8472 | 0.7525 | 0.3835 | |
|
|
| cosine_recall@150 | 0.8745 | 0.5825 | 0.5183 | 0.3287 | 0.8825 | 0.8026 | 0.4309 | |
|
|
| cosine_recall@200 | 0.9057 | 0.6147 | 0.5517 | 0.3631 | 0.9051 | 0.8418 | 0.4715 | |
|
|
| cosine_ndcg@1 | 0.6571 | 0.1243 | 0.2956 | 0.3495 | 0.4113 | 0.2943 | 0.0971 | |
|
|
| cosine_ndcg@20 | 0.6845 | 0.5385 | 0.4601 | 0.2468 | 0.5117 | 0.3919 | 0.1385 | |
|
|
| cosine_ndcg@50 | 0.704 | 0.5012 | 0.4229 | 0.2394 | 0.542 | 0.4256 | 0.1656 | |
|
|
| cosine_ndcg@100 | 0.7589 | 0.5147 | 0.4371 | 0.2619 | 0.5588 | 0.4462 | 0.1835 | |
|
|
| cosine_ndcg@150 | 0.7774 | 0.5348 | 0.4629 | 0.2787 | 0.5656 | 0.4561 | 0.1931 | |
|
|
| **cosine_ndcg@200** | **0.7893** | **0.5505** | **0.4797** | **0.2919** | **0.5697** | **0.4632** | **0.2007** | |
|
|
| cosine_mrr@1 | 0.6571 | 0.1243 | 0.2956 | 0.3495 | 0.4113 | 0.2943 | 0.0971 | |
|
|
| cosine_mrr@20 | 0.8103 | 0.5515 | 0.4896 | 0.4485 | 0.4979 | 0.3779 | 0.1522 | |
|
|
| cosine_mrr@50 | 0.8103 | 0.5515 | 0.4909 | 0.4515 | 0.501 | 0.3815 | 0.1564 | |
|
|
| cosine_mrr@100 | 0.8103 | 0.5515 | 0.4911 | 0.4519 | 0.5018 | 0.3827 | 0.158 | |
|
|
| cosine_mrr@150 | 0.8103 | 0.5515 | 0.4912 | 0.4523 | 0.5021 | 0.3831 | 0.1585 | |
|
|
| cosine_mrr@200 | 0.8103 | 0.5515 | 0.4912 | 0.4525 | 0.5021 | 0.3832 | 0.1588 | |
|
|
| cosine_map@1 | 0.6571 | 0.1243 | 0.2956 | 0.3495 | 0.4113 | 0.2943 | 0.0971 | |
|
|
| cosine_map@20 | 0.5418 | 0.4028 | 0.3236 | 0.147 | 0.4264 | 0.3097 | 0.0875 | |
|
|
| cosine_map@50 | 0.5327 | 0.3422 | 0.2644 | 0.1267 | 0.4338 | 0.3174 | 0.093 | |
|
|
| cosine_map@100 | 0.5657 | 0.3395 | 0.2576 | 0.1326 | 0.436 | 0.3199 | 0.095 | |
|
|
| cosine_map@150 | 0.5734 | 0.3478 | 0.2669 | 0.1352 | 0.4366 | 0.3207 | 0.0957 | |
|
|
| cosine_map@200 | 0.5772 | 0.3534 | 0.2722 | 0.1368 | 0.4368 | 0.3212 | 0.0961 | |
|
|
| cosine_map@500 | 0.5814 | 0.3631 | 0.2833 | 0.1407 | 0.4373 | 0.3219 | 0.0971 | |
|
|
|
|
|
<!-- |
|
|
## Bias, Risks and Limitations |
|
|
|
|
|
*What are the known or foreseeable issues stemming from this model? You could also flag here known failure cases or weaknesses of the model.* |
|
|
--> |
|
|
|
|
|
<!-- |
|
|
### Recommendations |
|
|
|
|
|
*What are recommendations with respect to the foreseeable issues? For example, filtering explicit content.* |
|
|
--> |
|
|
|
|
|
## Training Details |
|
|
|
|
|
### Training Datasets |
|
|
<details><summary>full_en</summary> |
|
|
|
|
|
#### full_en |
|
|
|
|
|
* Dataset: full_en |
|
|
* Size: 28,880 training samples |
|
|
* Columns: <code>anchor</code> and <code>positive</code> |
|
|
* Approximate statistics based on the first 1000 samples: |
|
|
| | anchor | positive | |
|
|
|:--------|:--------------------------------------------------------------------------------|:---------------------------------------------------------------------------------| |
|
|
| type | string | string | |
|
|
| details | <ul><li>min: 3 tokens</li><li>mean: 5.0 tokens</li><li>max: 10 tokens</li></ul> | <ul><li>min: 3 tokens</li><li>mean: 5.01 tokens</li><li>max: 13 tokens</li></ul> | |
|
|
* Samples: |
|
|
| anchor | positive | |
|
|
|:-----------------------------------------|:-----------------------------------------| |
|
|
| <code>air commodore</code> | <code>flight lieutenant</code> | |
|
|
| <code>command and control officer</code> | <code>flight officer</code> | |
|
|
| <code>air commodore</code> | <code>command and control officer</code> | |
|
|
* Loss: [<code>GISTEmbedLoss</code>](https://sbert.net/docs/package_reference/sentence_transformer/losses.html#gistembedloss) with these parameters: |
|
|
```json |
|
|
{'guide': SentenceTransformer( |
|
|
(0): Transformer({'max_seq_length': 128, 'do_lower_case': False}) with Transformer model: BertModel |
|
|
(1): Pooling({'word_embedding_dimension': 384, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': True, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True}) |
|
|
(2): Normalize() |
|
|
), 'temperature': 0.01, 'margin_strategy': 'absolute', 'margin': 0.0} |
|
|
``` |
|
|
</details> |
|
|
<details><summary>full_de</summary> |
|
|
|
|
|
#### full_de |
|
|
|
|
|
* Dataset: full_de |
|
|
* Size: 23,023 training samples |
|
|
* Columns: <code>anchor</code> and <code>positive</code> |
|
|
* Approximate statistics based on the first 1000 samples: |
|
|
| | anchor | positive | |
|
|
|:--------|:----------------------------------------------------------------------------------|:----------------------------------------------------------------------------------| |
|
|
| type | string | string | |
|
|
| details | <ul><li>min: 3 tokens</li><li>mean: 11.05 tokens</li><li>max: 45 tokens</li></ul> | <ul><li>min: 3 tokens</li><li>mean: 11.43 tokens</li><li>max: 45 tokens</li></ul> | |
|
|
* Samples: |
|
|
| anchor | positive | |
|
|
|:----------------------------------|:-----------------------------------------------------| |
|
|
| <code>Staffelkommandantin</code> | <code>Kommodore</code> | |
|
|
| <code>Luftwaffenoffizierin</code> | <code>Luftwaffenoffizier/Luftwaffenoffizierin</code> | |
|
|
| <code>Staffelkommandantin</code> | <code>Luftwaffenoffizierin</code> | |
|
|
* Loss: [<code>GISTEmbedLoss</code>](https://sbert.net/docs/package_reference/sentence_transformer/losses.html#gistembedloss) with these parameters: |
|
|
```json |
|
|
{'guide': SentenceTransformer( |
|
|
(0): Transformer({'max_seq_length': 128, 'do_lower_case': False}) with Transformer model: BertModel |
|
|
(1): Pooling({'word_embedding_dimension': 384, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': True, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True}) |
|
|
(2): Normalize() |
|
|
), 'temperature': 0.01, 'margin_strategy': 'absolute', 'margin': 0.0} |
|
|
``` |
|
|
</details> |
|
|
<details><summary>full_es</summary> |
|
|
|
|
|
#### full_es |
|
|
|
|
|
* Dataset: full_es |
|
|
* Size: 20,724 training samples |
|
|
* Columns: <code>anchor</code> and <code>positive</code> |
|
|
* Approximate statistics based on the first 1000 samples: |
|
|
| | anchor | positive | |
|
|
|:--------|:----------------------------------------------------------------------------------|:----------------------------------------------------------------------------------| |
|
|
| type | string | string | |
|
|
| details | <ul><li>min: 3 tokens</li><li>mean: 12.95 tokens</li><li>max: 50 tokens</li></ul> | <ul><li>min: 3 tokens</li><li>mean: 12.57 tokens</li><li>max: 50 tokens</li></ul> | |
|
|
* Samples: |
|
|
| anchor | positive | |
|
|
|:------------------------------------|:-------------------------------------------| |
|
|
| <code>jefe de escuadrón</code> | <code>instructor</code> | |
|
|
| <code>comandante de aeronave</code> | <code>instructor de simulador</code> | |
|
|
| <code>instructor</code> | <code>oficial del Ejército del Aire</code> | |
|
|
* Loss: [<code>GISTEmbedLoss</code>](https://sbert.net/docs/package_reference/sentence_transformer/losses.html#gistembedloss) with these parameters: |
|
|
```json |
|
|
{'guide': SentenceTransformer( |
|
|
(0): Transformer({'max_seq_length': 128, 'do_lower_case': False}) with Transformer model: BertModel |
|
|
(1): Pooling({'word_embedding_dimension': 384, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': True, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True}) |
|
|
(2): Normalize() |
|
|
), 'temperature': 0.01, 'margin_strategy': 'absolute', 'margin': 0.0} |
|
|
``` |
|
|
</details> |
|
|
<details><summary>full_zh</summary> |
|
|
|
|
|
#### full_zh |
|
|
|
|
|
* Dataset: full_zh |
|
|
* Size: 30,401 training samples |
|
|
* Columns: <code>anchor</code> and <code>positive</code> |
|
|
* Approximate statistics based on the first 1000 samples: |
|
|
| | anchor | positive | |
|
|
|:--------|:---------------------------------------------------------------------------------|:---------------------------------------------------------------------------------| |
|
|
| type | string | string | |
|
|
| details | <ul><li>min: 4 tokens</li><li>mean: 8.36 tokens</li><li>max: 20 tokens</li></ul> | <ul><li>min: 4 tokens</li><li>mean: 8.95 tokens</li><li>max: 27 tokens</li></ul> | |
|
|
* Samples: |
|
|
| anchor | positive | |
|
|
|:------------------|:---------------------| |
|
|
| <code>技术总监</code> | <code>技术和运营总监</code> | |
|
|
| <code>技术总监</code> | <code>技术主管</code> | |
|
|
| <code>技术总监</code> | <code>技术艺术总监</code> | |
|
|
* Loss: [<code>GISTEmbedLoss</code>](https://sbert.net/docs/package_reference/sentence_transformer/losses.html#gistembedloss) with these parameters: |
|
|
```json |
|
|
{'guide': SentenceTransformer( |
|
|
(0): Transformer({'max_seq_length': 128, 'do_lower_case': False}) with Transformer model: BertModel |
|
|
(1): Pooling({'word_embedding_dimension': 384, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': True, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True}) |
|
|
(2): Normalize() |
|
|
), 'temperature': 0.01, 'margin_strategy': 'absolute', 'margin': 0.0} |
|
|
``` |
|
|
</details> |
|
|
<details><summary>mix</summary> |
|
|
|
|
|
#### mix |
|
|
|
|
|
* Dataset: mix |
|
|
* Size: 21,760 training samples |
|
|
* Columns: <code>anchor</code> and <code>positive</code> |
|
|
* Approximate statistics based on the first 1000 samples: |
|
|
| | anchor | positive | |
|
|
|:--------|:---------------------------------------------------------------------------------|:----------------------------------------------------------------------------------| |
|
|
| type | string | string | |
|
|
| details | <ul><li>min: 2 tokens</li><li>mean: 5.65 tokens</li><li>max: 14 tokens</li></ul> | <ul><li>min: 2 tokens</li><li>mean: 10.08 tokens</li><li>max: 30 tokens</li></ul> | |
|
|
* Samples: |
|
|
| anchor | positive | |
|
|
|:------------------------------------------|:----------------------------------------------------------------| |
|
|
| <code>technical manager</code> | <code>Technischer Direktor für Bühne, Film und Fernsehen</code> | |
|
|
| <code>head of technical</code> | <code>directora técnica</code> | |
|
|
| <code>head of technical department</code> | <code>技术艺术总监</code> | |
|
|
* Loss: [<code>GISTEmbedLoss</code>](https://sbert.net/docs/package_reference/sentence_transformer/losses.html#gistembedloss) with these parameters: |
|
|
```json |
|
|
{'guide': SentenceTransformer( |
|
|
(0): Transformer({'max_seq_length': 128, 'do_lower_case': False}) with Transformer model: BertModel |
|
|
(1): Pooling({'word_embedding_dimension': 384, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': True, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True}) |
|
|
(2): Normalize() |
|
|
), 'temperature': 0.01, 'margin_strategy': 'absolute', 'margin': 0.0} |
|
|
``` |
|
|
</details> |
|
|
|
|
|
### Training Hyperparameters |
|
|
#### Non-Default Hyperparameters |
|
|
|
|
|
- `eval_strategy`: steps |
|
|
- `per_device_train_batch_size`: 128 |
|
|
- `per_device_eval_batch_size`: 128 |
|
|
- `gradient_accumulation_steps`: 2 |
|
|
- `num_train_epochs`: 5 |
|
|
- `warmup_ratio`: 0.05 |
|
|
- `log_on_each_node`: False |
|
|
- `fp16`: True |
|
|
- `dataloader_num_workers`: 4 |
|
|
- `ddp_find_unused_parameters`: True |
|
|
- `batch_sampler`: no_duplicates |
|
|
|
|
|
#### All Hyperparameters |
|
|
<details><summary>Click to expand</summary> |
|
|
|
|
|
- `overwrite_output_dir`: False |
|
|
- `do_predict`: False |
|
|
- `eval_strategy`: steps |
|
|
- `prediction_loss_only`: True |
|
|
- `per_device_train_batch_size`: 128 |
|
|
- `per_device_eval_batch_size`: 128 |
|
|
- `per_gpu_train_batch_size`: None |
|
|
- `per_gpu_eval_batch_size`: None |
|
|
- `gradient_accumulation_steps`: 2 |
|
|
- `eval_accumulation_steps`: None |
|
|
- `torch_empty_cache_steps`: None |
|
|
- `learning_rate`: 5e-05 |
|
|
- `weight_decay`: 0.0 |
|
|
- `adam_beta1`: 0.9 |
|
|
- `adam_beta2`: 0.999 |
|
|
- `adam_epsilon`: 1e-08 |
|
|
- `max_grad_norm`: 1.0 |
|
|
- `num_train_epochs`: 5 |
|
|
- `max_steps`: -1 |
|
|
- `lr_scheduler_type`: linear |
|
|
- `lr_scheduler_kwargs`: {} |
|
|
- `warmup_ratio`: 0.05 |
|
|
- `warmup_steps`: 0 |
|
|
- `log_level`: passive |
|
|
- `log_level_replica`: warning |
|
|
- `log_on_each_node`: False |
|
|
- `logging_nan_inf_filter`: True |
|
|
- `save_safetensors`: True |
|
|
- `save_on_each_node`: False |
|
|
- `save_only_model`: False |
|
|
- `restore_callback_states_from_checkpoint`: False |
|
|
- `no_cuda`: False |
|
|
- `use_cpu`: False |
|
|
- `use_mps_device`: False |
|
|
- `seed`: 42 |
|
|
- `data_seed`: None |
|
|
- `jit_mode_eval`: False |
|
|
- `use_ipex`: False |
|
|
- `bf16`: False |
|
|
- `fp16`: True |
|
|
- `fp16_opt_level`: O1 |
|
|
- `half_precision_backend`: auto |
|
|
- `bf16_full_eval`: False |
|
|
- `fp16_full_eval`: False |
|
|
- `tf32`: None |
|
|
- `local_rank`: 0 |
|
|
- `ddp_backend`: None |
|
|
- `tpu_num_cores`: None |
|
|
- `tpu_metrics_debug`: False |
|
|
- `debug`: [] |
|
|
- `dataloader_drop_last`: True |
|
|
- `dataloader_num_workers`: 4 |
|
|
- `dataloader_prefetch_factor`: None |
|
|
- `past_index`: -1 |
|
|
- `disable_tqdm`: False |
|
|
- `remove_unused_columns`: True |
|
|
- `label_names`: None |
|
|
- `load_best_model_at_end`: False |
|
|
- `ignore_data_skip`: False |
|
|
- `fsdp`: [] |
|
|
- `fsdp_min_num_params`: 0 |
|
|
- `fsdp_config`: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False} |
|
|
- `tp_size`: 0 |
|
|
- `fsdp_transformer_layer_cls_to_wrap`: None |
|
|
- `accelerator_config`: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None} |
|
|
- `deepspeed`: None |
|
|
- `label_smoothing_factor`: 0.0 |
|
|
- `optim`: adamw_torch |
|
|
- `optim_args`: None |
|
|
- `adafactor`: False |
|
|
- `group_by_length`: False |
|
|
- `length_column_name`: length |
|
|
- `ddp_find_unused_parameters`: True |
|
|
- `ddp_bucket_cap_mb`: None |
|
|
- `ddp_broadcast_buffers`: False |
|
|
- `dataloader_pin_memory`: True |
|
|
- `dataloader_persistent_workers`: False |
|
|
- `skip_memory_metrics`: True |
|
|
- `use_legacy_prediction_loop`: False |
|
|
- `push_to_hub`: False |
|
|
- `resume_from_checkpoint`: None |
|
|
- `hub_model_id`: None |
|
|
- `hub_strategy`: every_save |
|
|
- `hub_private_repo`: None |
|
|
- `hub_always_push`: False |
|
|
- `gradient_checkpointing`: False |
|
|
- `gradient_checkpointing_kwargs`: None |
|
|
- `include_inputs_for_metrics`: False |
|
|
- `include_for_metrics`: [] |
|
|
- `eval_do_concat_batches`: True |
|
|
- `fp16_backend`: auto |
|
|
- `push_to_hub_model_id`: None |
|
|
- `push_to_hub_organization`: None |
|
|
- `mp_parameters`: |
|
|
- `auto_find_batch_size`: False |
|
|
- `full_determinism`: False |
|
|
- `torchdynamo`: None |
|
|
- `ray_scope`: last |
|
|
- `ddp_timeout`: 1800 |
|
|
- `torch_compile`: False |
|
|
- `torch_compile_backend`: None |
|
|
- `torch_compile_mode`: None |
|
|
- `include_tokens_per_second`: False |
|
|
- `include_num_input_tokens_seen`: False |
|
|
- `neftune_noise_alpha`: None |
|
|
- `optim_target_modules`: None |
|
|
- `batch_eval_metrics`: False |
|
|
- `eval_on_start`: False |
|
|
- `use_liger_kernel`: False |
|
|
- `eval_use_gather_object`: False |
|
|
- `average_tokens_across_devices`: False |
|
|
- `prompts`: None |
|
|
- `batch_sampler`: no_duplicates |
|
|
- `multi_dataset_batch_sampler`: proportional |
|
|
|
|
|
</details> |
|
|
|
|
|
### Training Logs |
|
|
| Epoch | Step | Training Loss | full_en_cosine_ndcg@200 | full_es_cosine_ndcg@200 | full_de_cosine_ndcg@200 | full_zh_cosine_ndcg@200 | mix_es_cosine_ndcg@200 | mix_de_cosine_ndcg@200 | mix_zh_cosine_ndcg@200 | |
|
|
|:------:|:----:|:-------------:|:-----------------------:|:-----------------------:|:-----------------------:|:-----------------------:|:----------------------:|:----------------------:|:----------------------:| |
|
|
| -1 | -1 | - | 0.7322 | 0.4690 | 0.3853 | 0.2723 | 0.3209 | 0.2244 | 0.0919 | |
|
|
| 0.0021 | 1 | 23.8878 | - | - | - | - | - | - | - | |
|
|
| 0.2058 | 100 | 7.2098 | - | - | - | - | - | - | - | |
|
|
| 0.4115 | 200 | 4.2635 | 0.7800 | 0.5132 | 0.4268 | 0.2798 | 0.4372 | 0.2996 | 0.1447 | |
|
|
| 0.6173 | 300 | 4.1931 | - | - | - | - | - | - | - | |
|
|
| 0.8230 | 400 | 3.73 | 0.7863 | 0.5274 | 0.4451 | 0.2805 | 0.4762 | 0.3455 | 0.1648 | |
|
|
| 1.0309 | 500 | 3.3569 | - | - | - | - | - | - | - | |
|
|
| 1.2366 | 600 | 3.6464 | 0.7868 | 0.5372 | 0.4540 | 0.2813 | 0.5063 | 0.3794 | 0.1755 | |
|
|
| 1.4424 | 700 | 3.0772 | - | - | - | - | - | - | - | |
|
|
| 1.6481 | 800 | 3.114 | 0.7906 | 0.5391 | 0.4576 | 0.2832 | 0.5221 | 0.4047 | 0.1779 | |
|
|
| 1.8539 | 900 | 2.9246 | - | - | - | - | - | - | - | |
|
|
| 2.0617 | 1000 | 2.7479 | 0.7873 | 0.5423 | 0.4631 | 0.2871 | 0.5323 | 0.4143 | 0.1843 | |
|
|
| 2.2675 | 1100 | 3.049 | - | - | - | - | - | - | - | |
|
|
| 2.4733 | 1200 | 2.6137 | 0.7878 | 0.5418 | 0.4685 | 0.2870 | 0.5470 | 0.4339 | 0.1932 | |
|
|
| 2.6790 | 1300 | 2.8607 | - | - | - | - | - | - | - | |
|
|
| 2.8848 | 1400 | 2.7071 | 0.7889 | 0.5465 | 0.4714 | 0.2891 | 0.5504 | 0.4362 | 0.1944 | |
|
|
| 3.0926 | 1500 | 2.7012 | - | - | - | - | - | - | - | |
|
|
| 3.2984 | 1600 | 2.7423 | 0.7882 | 0.5471 | 0.4748 | 0.2868 | 0.5542 | 0.4454 | 0.1976 | |
|
|
| 3.5041 | 1700 | 2.5316 | - | - | - | - | - | - | - | |
|
|
| 3.7099 | 1800 | 2.6344 | 0.7900 | 0.5498 | 0.4763 | 0.2857 | 0.5639 | 0.4552 | 0.1954 | |
|
|
| 3.9156 | 1900 | 2.4983 | - | - | - | - | - | - | - | |
|
|
| 4.1235 | 2000 | 2.5423 | 0.7894 | 0.5499 | 0.4786 | 0.2870 | 0.5644 | 0.4576 | 0.1974 | |
|
|
| 4.3292 | 2100 | 2.5674 | - | - | - | - | - | - | - | |
|
|
| 4.5350 | 2200 | 2.6237 | 0.7899 | 0.5502 | 0.4802 | 0.2843 | 0.5674 | 0.4607 | 0.1993 | |
|
|
| 4.7407 | 2300 | 2.3776 | - | - | - | - | - | - | - | |
|
|
| 4.9465 | 2400 | 2.1116 | 0.7893 | 0.5505 | 0.4797 | 0.2919 | 0.5697 | 0.4632 | 0.2007 | |
|
|
|
|
|
|
|
|
### Framework Versions |
|
|
- Python: 3.11.11 |
|
|
- Sentence Transformers: 4.1.0 |
|
|
- Transformers: 4.51.3 |
|
|
- PyTorch: 2.6.0+cu124 |
|
|
- Accelerate: 1.6.0 |
|
|
- Datasets: 3.5.0 |
|
|
- Tokenizers: 0.21.1 |
|
|
|
|
|
## Citation |
|
|
|
|
|
### BibTeX |
|
|
|
|
|
#### Sentence Transformers |
|
|
```bibtex |
|
|
@inproceedings{reimers-2019-sentence-bert, |
|
|
title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks", |
|
|
author = "Reimers, Nils and Gurevych, Iryna", |
|
|
booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing", |
|
|
month = "11", |
|
|
year = "2019", |
|
|
publisher = "Association for Computational Linguistics", |
|
|
url = "https://arxiv.org/abs/1908.10084", |
|
|
} |
|
|
``` |
|
|
|
|
|
#### GISTEmbedLoss |
|
|
```bibtex |
|
|
@misc{solatorio2024gistembed, |
|
|
title={GISTEmbed: Guided In-sample Selection of Training Negatives for Text Embedding Fine-tuning}, |
|
|
author={Aivin V. Solatorio}, |
|
|
year={2024}, |
|
|
eprint={2402.16829}, |
|
|
archivePrefix={arXiv}, |
|
|
primaryClass={cs.LG} |
|
|
} |
|
|
``` |
|
|
|
|
|
<!-- |
|
|
## Glossary |
|
|
|
|
|
*Clearly define terms in order to be accessible across audiences.* |
|
|
--> |
|
|
|
|
|
<!-- |
|
|
## Model Card Authors |
|
|
|
|
|
*Lists the people who create the model card, providing recognition and accountability for the detailed work that goes into its construction.* |
|
|
--> |
|
|
|
|
|
<!-- |
|
|
## Model Card Contact |
|
|
|
|
|
*Provides a way for people who have updates to the Model Card, suggestions, or questions, to contact the Model Card authors.* |
|
|
--> |
