potsu-potsu/bge-base-biomedical-matryoshka-v2

BGE Base Biomedical MRL

This is a sentence-transformers model finetuned from BAAI/bge-base-en-v1.5 on the json dataset. It maps sentences & paragraphs to a 768-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more.

Model Details

Model Description

• Model Type: Sentence Transformer
• Base model: BAAI/bge-base-en-v1.5
• Maximum Sequence Length: 512 tokens
• Output Dimensionality: 768 dimensions
• Similarity Function: Cosine Similarity
• Training Dataset:
  • json
• Language: en
• License: apache-2.0

Model Sources

• Documentation: Sentence Transformers Documentation
• Repository: Sentence Transformers on GitHub
• Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'max_seq_length': 512, 'do_lower_case': True}) with Transformer model: BertModel 
  (1): Pooling({'word_embedding_dimension': 768, 'pooling_mode_cls_token': True, 'pooling_mode_mean_tokens': False, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
  (2): Normalize()
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("potsu-potsu/bge-base-biomedical-matryoshka-v2")
# Run inference
sentences = [
    'What are the effects of the deletion of all three Pcdh clusters (tricluster deletion) in mice?',
    'Multicluster Pcdh diversity is required for mouse olfactory neural circuit assembly. The vertebrate clustered protocadherin (Pcdh) cell surface proteins are encoded by three closely linked gene clusters (Pcdhα, Pcdhβ, and Pcdhγ). Although deletion of individual Pcdh clusters had subtle phenotypic consequences, the loss of all three clusters (tricluster deletion) led to a severe axonal arborization defect and loss of self-avoidance.',
    'Investigators proposed that there have been three extended periods in the evolution of gene regulatory elements. Early vertebrate evolution was characterized by regulatory gains near transcription factors and developmental genes, but this trend was replaced by innovations near extracellular signaling genes, and then innovations near posttranslational protein modifiers.',
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 768]

# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities.shape)
# [3, 3]

Evaluation

Metrics

Information Retrieval

• Dataset: dim_768
• Evaluated with InformationRetrievalEvaluator with these parameters:

  {
      "truncate_dim": 768
  }
  

Metric	Value
cosine_accuracy@1	0.7199
cosine_accuracy@3	0.8487
cosine_accuracy@5	0.8812
cosine_accuracy@10	0.9109
cosine_precision@1	0.7199
cosine_precision@3	0.5837
cosine_precision@5	0.5035
cosine_precision@10	0.3898
cosine_recall@1	0.2255
cosine_recall@3	0.3824
cosine_recall@5	0.4676
cosine_recall@10	0.5895
cosine_ndcg@10	0.6739
cosine_mrr@10	0.7897
cosine_map@100	0.6022

Information Retrieval

• Dataset: dim_512
• Evaluated with InformationRetrievalEvaluator with these parameters:

  {
      "truncate_dim": 512
  }
  

Metric	Value
cosine_accuracy@1	0.7199
cosine_accuracy@3	0.8416
cosine_accuracy@5	0.8784
cosine_accuracy@10	0.9137
cosine_precision@1	0.7199
cosine_precision@3	0.5771
cosine_precision@5	0.499
cosine_precision@10	0.3893
cosine_recall@1	0.224
cosine_recall@3	0.3798
cosine_recall@5	0.4634
cosine_recall@10	0.5904
cosine_ndcg@10	0.6717
cosine_mrr@10	0.7864
cosine_map@100	0.5974

Information Retrieval

• Dataset: dim_256
• Evaluated with InformationRetrievalEvaluator with these parameters:

  {
      "truncate_dim": 256
  }
  

Metric	Value
cosine_accuracy@1	0.7199
cosine_accuracy@3	0.8345
cosine_accuracy@5	0.8685
cosine_accuracy@10	0.9052
cosine_precision@1	0.7199
cosine_precision@3	0.5672
cosine_precision@5	0.4908
cosine_precision@10	0.3832
cosine_recall@1	0.2251
cosine_recall@3	0.3696
cosine_recall@5	0.454
cosine_recall@10	0.5767
cosine_ndcg@10	0.6614
cosine_mrr@10	0.7821
cosine_map@100	0.585

Information Retrieval

• Dataset: dim_128
• Evaluated with InformationRetrievalEvaluator with these parameters:

  {
      "truncate_dim": 128
  }
  

Metric	Value
cosine_accuracy@1	0.7044
cosine_accuracy@3	0.8076
cosine_accuracy@5	0.8515
cosine_accuracy@10	0.8868
cosine_precision@1	0.7044
cosine_precision@3	0.5512
cosine_precision@5	0.4676
cosine_precision@10	0.3642
cosine_recall@1	0.2168
cosine_recall@3	0.3559
cosine_recall@5	0.4302
cosine_recall@10	0.5439
cosine_ndcg@10	0.6335
cosine_mrr@10	0.7664
cosine_map@100	0.5528

Information Retrieval

• Dataset: dim_64
• Evaluated with InformationRetrievalEvaluator with these parameters:

  {
      "truncate_dim": 64
  }
  

Metric	Value
cosine_accuracy@1	0.6535
cosine_accuracy@3	0.7808
cosine_accuracy@5	0.8232
cosine_accuracy@10	0.8614
cosine_precision@1	0.6535
cosine_precision@3	0.5177
cosine_precision@5	0.4379
cosine_precision@10	0.3335
cosine_recall@1	0.1965
cosine_recall@3	0.3309
cosine_recall@5	0.3971
cosine_recall@10	0.4998
cosine_ndcg@10	0.5832
cosine_mrr@10	0.7249
cosine_map@100	0.4959

Training Details

Training Dataset

json

• Dataset: json
• Size: 4,012 training samples
• Columns: anchor and positive
• Approximate statistics based on the first 1000 samples:

  	anchor	positive
  type	string	string
  details	min: 5 tokens mean: 16.13 tokens max: 49 tokens	min: 3 tokens mean: 63.38 tokens max: 485 tokens

• Samples:

  anchor	positive
  What is the implication of histone lysine methylation in medulloblastoma?	Aberrant patterns of H3K4, H3K9, and H3K27 histone lysine methylation were shown to result in histone code alterations, which induce changes in gene expression, and affect the proliferation rate of cells in medulloblastoma.
  What is the role of STAG1/STAG2 proteins in differentiation?	STAG1/STAG2 proteins are tumour suppressor proteins that suppress cell proliferation and are essential for differentiation.
  What is the association between cell phone use and glioblastoma?	The association between cell phone use and incident glioblastoma remains unclear. Some studies have reported that cell phone use was associated with incident glioblastoma, and with reduced survival of patients diagnosed with glioblastoma. However, other studies have repeatedly replicated to find an association between cell phone use and glioblastoma.

• Loss: MatryoshkaLoss with these parameters:

  {
      "loss": "MultipleNegativesRankingLoss",
      "matryoshka_dims": [
          768,
          512,
          256,
          128,
          64
      ],
      "matryoshka_weights": [
          1,
          1,
          1,
          1,
          1
      ],
      "n_dims_per_step": -1
  }
  

Training Hyperparameters

Non-Default Hyperparameters

• eval_strategy: epoch
• per_device_train_batch_size: 32
• per_device_eval_batch_size: 16
• learning_rate: 2e-05
• num_train_epochs: 4
• lr_scheduler_type: cosine
• warmup_ratio: 0.1
• bf16: True
• tf32: True
• load_best_model_at_end: True
• optim: adamw_torch_fused
• batch_sampler: no_duplicates

All Hyperparameters

Click to expand

• overwrite_output_dir: False
• do_predict: False
• eval_strategy: epoch
• prediction_loss_only: True
• per_device_train_batch_size: 32
• per_device_eval_batch_size: 16
• per_gpu_train_batch_size: None
• per_gpu_eval_batch_size: None
• gradient_accumulation_steps: 1
• eval_accumulation_steps: None
• torch_empty_cache_steps: None
• learning_rate: 2e-05
• weight_decay: 0.0
• adam_beta1: 0.9
• adam_beta2: 0.999
• adam_epsilon: 1e-08
• max_grad_norm: 1.0
• num_train_epochs: 4
• max_steps: -1
• lr_scheduler_type: cosine
• lr_scheduler_kwargs: {}
• warmup_ratio: 0.1
• warmup_steps: 0
• log_level: passive
• log_level_replica: warning
• log_on_each_node: True
• logging_nan_inf_filter: True
• save_safetensors: True
• save_on_each_node: False
• save_only_model: False
• restore_callback_states_from_checkpoint: False
• no_cuda: False
• use_cpu: False
• use_mps_device: False
• seed: 42
• data_seed: None
• jit_mode_eval: False
• use_ipex: False
• bf16: True
• fp16: False
• fp16_opt_level: O1
• half_precision_backend: auto
• bf16_full_eval: False
• fp16_full_eval: False
• tf32: True
• local_rank: 0
• ddp_backend: None
• tpu_num_cores: None
• tpu_metrics_debug: False
• debug: []
• dataloader_drop_last: False
• dataloader_num_workers: 0
• dataloader_prefetch_factor: None
• past_index: -1
• disable_tqdm: False
• remove_unused_columns: True
• label_names: None
• load_best_model_at_end: True
• ignore_data_skip: False
• fsdp: []
• fsdp_min_num_params: 0
• fsdp_config: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
• fsdp_transformer_layer_cls_to_wrap: None
• accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
• deepspeed: None
• label_smoothing_factor: 0.0
• optim: adamw_torch_fused
• optim_args: None
• adafactor: False
• group_by_length: False
• length_column_name: length
• ddp_find_unused_parameters: None
• ddp_bucket_cap_mb: None
• ddp_broadcast_buffers: False
• dataloader_pin_memory: True
• dataloader_persistent_workers: False
• skip_memory_metrics: True
• use_legacy_prediction_loop: False
• push_to_hub: False
• resume_from_checkpoint: None
• hub_model_id: None
• hub_strategy: every_save
• hub_private_repo: None
• hub_always_push: False
• gradient_checkpointing: False
• gradient_checkpointing_kwargs: None
• include_inputs_for_metrics: False
• include_for_metrics: []
• eval_do_concat_batches: True
• fp16_backend: auto
• push_to_hub_model_id: None
• push_to_hub_organization: None
• mp_parameters:
• auto_find_batch_size: False
• full_determinism: False
• torchdynamo: None
• ray_scope: last
• ddp_timeout: 1800
• torch_compile: False
• torch_compile_backend: None
• torch_compile_mode: None
• include_tokens_per_second: False
• include_num_input_tokens_seen: False
• neftune_noise_alpha: None
• optim_target_modules: None
• batch_eval_metrics: False
• eval_on_start: False
• use_liger_kernel: False
• eval_use_gather_object: False
• average_tokens_across_devices: False
• prompts: None
• batch_sampler: no_duplicates
• multi_dataset_batch_sampler: proportional

Training Logs

Epoch	Step	Training Loss	dim_768_cosine_ndcg@10	dim_512_cosine_ndcg@10	dim_256_cosine_ndcg@10	dim_128_cosine_ndcg@10	dim_64_cosine_ndcg@10
0.0794	10	2.1512	-	-	-	-	-
0.1587	20	1.8449	-	-	-	-	-
0.2381	30	1.2477	-	-	-	-	-
0.3175	40	1.3083	-	-	-	-	-
0.3968	50	0.7793	-	-	-	-	-
0.4762	60	0.7176	-	-	-	-	-
0.5556	70	1.0079	-	-	-	-	-
0.6349	80	0.6972	-	-	-	-	-
0.7143	90	0.5267	-	-	-	-	-
0.7937	100	0.929	-	-	-	-	-
0.8730	110	0.5549	-	-	-	-	-
0.9524	120	0.4972	-	-	-	-	-
1.0	126	-	0.6891	0.6874	0.6711	0.6336	0.5709
1.0317	130	0.7263	-	-	-	-	-
1.1111	140	0.2828	-	-	-	-	-
1.1905	150	0.3925	-	-	-	-	-
1.2698	160	0.4131	-	-	-	-	-
1.3492	170	0.2274	-	-	-	-	-
1.4286	180	0.4237	-	-	-	-	-
1.5079	190	0.343	-	-	-	-	-
1.5873	200	0.4364	-	-	-	-	-
1.6667	210	0.3228	-	-	-	-	-
1.7460	220	0.2519	-	-	-	-	-
1.8254	230	0.2613	-	-	-	-	-
1.9048	240	0.2888	-	-	-	-	-
1.9841	250	0.3633	-	-	-	-	-
2.0	252	-	0.6825	0.6804	0.6668	0.6371	0.583
2.0635	260	0.1836	-	-	-	-	-
2.1429	270	0.2064	-	-	-	-	-
2.2222	280	0.1623	-	-	-	-	-
2.3016	290	0.1402	-	-	-	-	-
2.3810	300	0.1687	-	-	-	-	-
2.4603	310	0.1391	-	-	-	-	-
2.5397	320	0.138	-	-	-	-	-
2.6190	330	0.1198	-	-	-	-	-
2.6984	340	0.1717	-	-	-	-	-
2.7778	350	0.1106	-	-	-	-	-
2.8571	360	0.0958	-	-	-	-	-
2.9365	370	0.1679	-	-	-	-	-
3.0	378	-	0.6747	0.6738	0.6621	0.6342	0.5837
3.0159	380	0.2262	-	-	-	-	-
3.0952	390	0.1048	-	-	-	-	-
3.1746	400	0.0638	-	-	-	-	-
3.2540	410	0.1694	-	-	-	-	-
3.3333	420	0.1466	-	-	-	-	-
3.4127	430	0.1342	-	-	-	-	-
3.4921	440	0.1044	-	-	-	-	-
3.5714	450	0.1333	-	-	-	-	-
3.6508	460	0.1075	-	-	-	-	-
3.7302	470	0.0525	-	-	-	-	-
3.8095	480	0.1423	-	-	-	-	-
3.8889	490	0.0716	-	-	-	-	-
3.9683	500	0.0813	-	-	-	-	-
4.0	504	-	0.6739	0.6717	0.6614	0.6335	0.5832

• The bold row denotes the saved checkpoint.

Framework Versions

• Python: 3.12.5
• Sentence Transformers: 4.1.0
• Transformers: 4.52.4
• PyTorch: 2.7.1+cu128
• Accelerate: 1.7.0
• Datasets: 3.6.0
• Tokenizers: 0.21.1

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

MatryoshkaLoss

@misc{kusupati2024matryoshka,
    title={Matryoshka Representation Learning},
    author={Aditya Kusupati and Gantavya Bhatt and Aniket Rege and Matthew Wallingford and Aditya Sinha and Vivek Ramanujan and William Howard-Snyder and Kaifeng Chen and Sham Kakade and Prateek Jain and Ali Farhadi},
    year={2024},
    eprint={2205.13147},
    archivePrefix={arXiv},
    primaryClass={cs.LG}
}

MultipleNegativesRankingLoss

@misc{henderson2017efficient,
    title={Efficient Natural Language Response Suggestion for Smart Reply},
    author={Matthew Henderson and Rami Al-Rfou and Brian Strope and Yun-hsuan Sung and Laszlo Lukacs and Ruiqi Guo and Sanjiv Kumar and Balint Miklos and Ray Kurzweil},
    year={2017},
    eprint={1705.00652},
    archivePrefix={arXiv},
    primaryClass={cs.CL}
}