manupande21/all-MiniLM-L6-v2-LoRA-finetuned-1M

SentenceTransformer based on sentence-transformers/all-MiniLM-L6-v2

This is a sentence-transformers model finetuned from sentence-transformers/all-MiniLM-L6-v2. It maps sentences & paragraphs to a 384-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more.

Model Details

Model Description

• Model Type: Sentence Transformer
• Base model: sentence-transformers/all-MiniLM-L6-v2
• Maximum Sequence Length: 512 tokens
• Output Dimensionality: 384 dimensions
• Similarity Function: Cosine Similarity

Model Sources

• Documentation: Sentence Transformers Documentation
• Repository: Sentence Transformers on GitHub
• Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'max_seq_length': 512, 'do_lower_case': False}) with Transformer model: PeftModelForFeatureExtraction 
  (1): Pooling({'word_embedding_dimension': 384, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': True, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("manupande21/all-MiniLM-L6-v2-LoRA-finetuned-1M")
# Run inference
sentences = [
    'what is the purpose of bands with braces',
    'The Purpose of Rubber Bands for Braces. If you have gotten a professional consultation from an orthodontist, you may have been given rubber bands for braces; lots of doctors recommend using rubber bands in conjunction with braces to help straighten teeth quicker.',
    'To successfully complete your orthodontic treatment plan, patients must work together with the orthodontist. The teeth and jaws can only move toward their corrected positions if the patient consistently wears the elastics (rubber bands), headgear or other appliances as prescribed. The following paragraphs describe the types of appliances that may be used during your treatment. Elastics (Rubber Bands) Wearing elastics (rubber bands) is the main way that Orthodontic treatment can improve the fit of your bite. Braces will straighten your teeth, but are unable to correct bite problems unless used in conjunction with elastics.',
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 384]

# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities.shape)
# [3, 3]

Evaluation

Metrics

Triplet

• Dataset: test-eval
• Evaluated with TripletEvaluator

Metric	Value
cosine_accuracy	0.9559

Training Details

Training Dataset

Unnamed Dataset

• Size: 800,000 training samples
• Columns: sentence_0, sentence_1, and sentence_2
• Approximate statistics based on the first 1000 samples:

  	sentence_0	sentence_1	sentence_2
  type	string	string	string
  details	min: 4 tokens mean: 9.05 tokens max: 38 tokens	min: 15 tokens mean: 80.09 tokens max: 232 tokens	min: 21 tokens mean: 78.11 tokens max: 192 tokens

• Samples:

  sentence_0	sentence_1	sentence_2
  american modernism is characterized by _____.	FROM MODERNISM TO POST-MODERNISM. POST-WAR ART IN AMERICA. After the Second World War, the art world was characterized by “triumphalism” in New York and a feeling of having won, not just a military war but also a cultural war. The French and their School of Paris had been routed. Also defeated was American Scene painting and its nativist illustrations of a naïve nation.	noun. 1 a disease of sheep characterized by an unsteady gait and staggering, caused by infestation of the brain with tapeworms (Taenia caenuris).oun. 1 a disease of sheep characterized by an unsteady gait and staggering, caused by infestation of the brain with tapeworms (Taenia caenuris).
  where was sen tom cotton born	Written By: born. Tom Cotton, in full Thomas Bryant Cotton (born May 13, 1977, Dardanelle, Arkansas, U.S.), American politician who was elected to the U.S. Senate as a Republican in 2014 and began his first term representing Arkansas the following year. He previously was a member of the U.S. House of Representatives (2013–15).	Cotton Facts: 1 The Cotton Belt spans the southern half of the United States, stretching from Virginia to California. 2 Cotton production covers more than 14 million acres or about 22,000 square miles of the United States.3 Texas is the leading cotton-producing state, producing about 4.5 million bales of cotton a year.4 Cotton contributes over $1 5 ... Tex Cotton production covers more than 14 million acres or about 22,000 square miles of the United States. 2 Texas is the leading cotton-producing state, producing about 4.5 million bales of cotton a year. 3 Cotton contributes over $1 billion to the Texas economy, ranking only behind the beef industry in total cash receipts.
  what is ginger beer	Gosling's Ginger Beer. Crabbie’s Alcoholic Ginger Beer was first crafted in 1801. Ginger beer is carbonated, sweetened beverage produced in two versions: alcoholic brewed ginger beer (which includes home-brewed) or a carbonated soft drink flavored primarily with ginger and sweetened with sugar or artificial sweeteners.oday, ginger beer is usually produced as a soft drink. Ginger beer and ginger ale as soft drinks have been moderately popular in many parts of the world since they were introduced. The similarities and differences between ginger ale and ginger beer are discussed in the history section of the ginger ale article.	Ginger is a commonly used flavoring agent and food product. Ginger is also available as an herbal supplement. The information contained in this leaflet refers to the use of ginger as an herbal supplement.

• Loss: TripletLoss with these parameters:

  {
      "distance_metric": "TripletDistanceMetric.EUCLIDEAN",
      "triplet_margin": 5
  }
  

Training Hyperparameters

Non-Default Hyperparameters

• eval_strategy: steps
• per_device_train_batch_size: 512
• per_device_eval_batch_size: 512
• fp16: True
• multi_dataset_batch_sampler: round_robin

All Hyperparameters

Click to expand

• overwrite_output_dir: False
• do_predict: False
• eval_strategy: steps
• prediction_loss_only: True
• per_device_train_batch_size: 512
• per_device_eval_batch_size: 512
• per_gpu_train_batch_size: None
• per_gpu_eval_batch_size: None
• gradient_accumulation_steps: 1
• eval_accumulation_steps: None
• learning_rate: 5e-05
• weight_decay: 0.0
• adam_beta1: 0.9
• adam_beta2: 0.999
• adam_epsilon: 1e-08
• max_grad_norm: 1
• num_train_epochs: 3
• max_steps: -1
• lr_scheduler_type: linear
• lr_scheduler_kwargs: {}
• warmup_ratio: 0.0
• warmup_steps: 0
• log_level: passive
• log_level_replica: warning
• log_on_each_node: True
• logging_nan_inf_filter: True
• save_safetensors: True
• save_on_each_node: False
• save_only_model: False
• restore_callback_states_from_checkpoint: False
• no_cuda: False
• use_cpu: False
• use_mps_device: False
• seed: 42
• data_seed: None
• jit_mode_eval: False
• use_ipex: False
• bf16: False
• fp16: True
• fp16_opt_level: O1
• half_precision_backend: auto
• bf16_full_eval: False
• fp16_full_eval: False
• tf32: None
• local_rank: 0
• ddp_backend: None
• tpu_num_cores: None
• tpu_metrics_debug: False
• debug: []
• dataloader_drop_last: False
• dataloader_num_workers: 0
• dataloader_prefetch_factor: None
• past_index: -1
• disable_tqdm: False
• remove_unused_columns: True
• label_names: None
• load_best_model_at_end: False
• ignore_data_skip: False
• fsdp: []
• fsdp_min_num_params: 0
• fsdp_config: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
• fsdp_transformer_layer_cls_to_wrap: None
• accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
• deepspeed: None
• label_smoothing_factor: 0.0
• optim: adamw_torch
• optim_args: None
• adafactor: False
• group_by_length: False
• length_column_name: length
• ddp_find_unused_parameters: None
• ddp_bucket_cap_mb: None
• ddp_broadcast_buffers: False
• dataloader_pin_memory: True
• dataloader_persistent_workers: False
• skip_memory_metrics: True
• use_legacy_prediction_loop: False
• push_to_hub: False
• resume_from_checkpoint: None
• hub_model_id: None
• hub_strategy: every_save
• hub_private_repo: False
• hub_always_push: False
• gradient_checkpointing: False
• gradient_checkpointing_kwargs: None
• include_inputs_for_metrics: False
• eval_do_concat_batches: True
• fp16_backend: auto
• push_to_hub_model_id: None
• push_to_hub_organization: None
• mp_parameters:
• auto_find_batch_size: False
• full_determinism: False
• torchdynamo: None
• ray_scope: last
• ddp_timeout: 1800
• torch_compile: False
• torch_compile_backend: None
• torch_compile_mode: None
• dispatch_batches: None
• split_batches: None
• include_tokens_per_second: False
• include_num_input_tokens_seen: False
• neftune_noise_alpha: None
• optim_target_modules: None
• batch_eval_metrics: False
• prompts: None
• batch_sampler: batch_sampler
• multi_dataset_batch_sampler: round_robin

Training Logs

Epoch	Step	Training Loss	test-eval_cosine_accuracy
0.3199	500	3.4281	-
0.6398	1000	1.7972	0.9516
0.9597	1500	1.5857	-
1.0	1563	-	0.9536
1.2796	2000	1.5243	0.9542
1.5995	2500	1.481	-
1.9194	3000	1.4559	0.9553
2.0	3126	-	0.9559

Framework Versions

• Python: 3.11.5
• Sentence Transformers: 4.1.0
• Transformers: 4.41.0
• PyTorch: 2.7.0+cu126
• Accelerate: 1.7.0
• Datasets: 3.2.0
• Tokenizers: 0.19.1

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

TripletLoss

@misc{hermans2017defense,
    title={In Defense of the Triplet Loss for Person Re-Identification},
    author={Alexander Hermans and Lucas Beyer and Bastian Leibe},
    year={2017},
    eprint={1703.07737},
    archivePrefix={arXiv},
    primaryClass={cs.CV}
}