Hugging Face's logo

Hulyyy
/
req-quality-setfit-128

Text Classification
setfit
Safetensors
sentence-transformers
bert
generated_from_setfit_trainer
Eval Results
Model card Files
xet
Community
req-quality-setfit-128 / README.md
Hulyyy's picture
Hulyyy
Push model using huggingface_hub.
5cd8935 verified
preview code
|
raw
history blame contribute delete
28.9 kB
metadata
tags:
  - setfit
  - sentence-transformers
  - text-classification
  - generated_from_setfit_trainer
widget:
  - text: >-
      The TCS software shall have an open architecture and be capable of being
      hosted on computers that are typically supported by the using Service.
  - text: >-
      3 It shall be possible to deregister up to ten functional numbers to items
      of equipment physically connected to the Cab radio within 30 seconds. (M)
  - text: >-
      1 The EIRENE system shall enable users to originate and receive calls by
      functional number. (M)
  - text: The product shall store new conference rooms.
  - text: >-
      Before authomatic transition to Shunting, ETCS shall request confirmation
      from the driver.
metrics:
  - micro_f1
  - macro_f1
  - hamming_accuracy
  - subset_accuracy
pipeline_tag: text-classification
library_name: setfit
inference: true
model-index:
  - name: SetFit
    results:
      - task:
          type: text-classification
          name: Text Classification
        dataset:
          name: Unknown
          type: unknown
          split: test
        metrics:
          - type: micro_f1
            value: 0.5631067961165048
            name: Micro_F1
          - type: macro_f1
            value: 0.5741774726671751
            name: Macro_F1
          - type: hamming_accuracy
            value: 0.8888888888888888
            name: Hamming_Accuracy
          - type: subset_accuracy
            value: 0.5637860082304527
            name: Subset_Accuracy

SetFit

This is a SetFit model that can be used for Text Classification. A WeightedBinaryRelevanceHead instance is used for classification.

The model has been trained using an efficient few-shot learning technique that involves:

  1. Fine-tuning a Sentence Transformer with contrastive learning.
  2. Training a classification head with features from the fine-tuned Sentence Transformer.

Model Details

Model Description

  • Model Type: SetFit
  • Classification head: a WeightedBinaryRelevanceHead instance
  • Maximum Sequence Length: 256 tokens

Model Sources

Evaluation

Metrics

Label Micro_F1 Macro_F1 Hamming_Accuracy Subset_Accuracy
all 0.5631 0.5742 0.8889 0.5638

Uses

Direct Use for Inference

First install the SetFit library:

pip install setfit

Then you can load this model and run inference.

from setfit import SetFitModel

# Download from the 🤗 Hub
model = SetFitModel.from_pretrained("Hulyyy/req-quality-setfit-128")
# Run inference
preds = model("The product shall store new conference rooms.")

Training Details

Training Set Metrics

Training set Min Median Max
Word count 2 19.3840 32

Training Hyperparameters

  • batch_size: (64, 64)
  • num_epochs: (3, 3)
  • max_steps: -1
  • sampling_strategy: oversampling
  • num_iterations: 200
  • body_learning_rate: (1e-05, 1e-05)
  • head_learning_rate: 1e-05
  • loss: CosineSimilarityLoss
  • distance_metric: cosine_distance
  • margin: 0.25
  • end_to_end: False
  • use_amp: True
  • warmup_proportion: 0.1
  • l2_weight: 0.01
  • seed: 42
  • eval_max_steps: -1
  • load_best_model_at_end: False

Training Results

Epoch Step Training Loss Validation Loss
0.0001 1 0.3722 -
0.0070 50 0.3592 -
0.0141 100 0.3552 -
0.0211 150 0.3271 -
0.0281 200 0.3016 -
0.0351 250 0.2783 -
0.0422 300 0.2642 -
0.0492 350 0.2587 -
0.0562 400 0.2568 -
0.0633 450 0.257 -
0.0703 500 0.2498 -
0.0773 550 0.2444 -
0.0844 600 0.2437 -
0.0914 650 0.2399 -
0.0984 700 0.2371 -
0.1054 750 0.2302 -
0.1125 800 0.2291 -
0.1195 850 0.2233 -
0.1265 900 0.2192 -
0.1336 950 0.2229 -
0.1406 1000 0.218 -
0.1476 1050 0.2096 -
0.1546 1100 0.2107 -
0.1617 1150 0.202 -
0.1687 1200 0.2009 -
0.1757 1250 0.1975 -
0.1828 1300 0.1969 -
0.1898 1350 0.1955 -
0.1968 1400 0.1871 -
0.2039 1450 0.1837 -
0.2109 1500 0.1732 -
0.2179 1550 0.1702 -
0.2249 1600 0.1687 -
0.2320 1650 0.16 -
0.2390 1700 0.1551 -
0.2460 1750 0.1495 -
0.2531 1800 0.1386 -
0.2601 1850 0.1339 -
0.2671 1900 0.1263 -
0.2741 1950 0.1232 -
0.2812 2000 0.1209 -
0.2882 2050 0.112 -
0.2952 2100 0.0995 -
0.3023 2150 0.0932 -
0.3093 2200 0.0869 -
0.3163 2250 0.076 -
0.3234 2300 0.0656 -
0.3304 2350 0.067 -
0.3374 2400 0.0602 -
0.3444 2450 0.0561 -
0.3515 2500 0.0532 -
0.3585 2550 0.0465 -
0.3655 2600 0.0443 -
0.3726 2650 0.0443 -
0.3796 2700 0.0435 -
0.3866 2750 0.0392 -
0.3936 2800 0.0417 -
0.4007 2850 0.0354 -
0.4077 2900 0.0379 -
0.4147 2950 0.037 -
0.4218 3000 0.0366 -
0.4288 3050 0.0328 -
0.4358 3100 0.032 -
0.4429 3150 0.027 -
0.4499 3200 0.0329 -
0.4569 3250 0.026 -
0.4639 3300 0.0284 -
0.4710 3350 0.0274 -
0.4780 3400 0.0264 -
0.4850 3450 0.0239 -
0.4921 3500 0.0256 -
0.4991 3550 0.0256 -
0.5061 3600 0.024 -
0.5131 3650 0.0223 -
0.5202 3700 0.0225 -
0.5272 3750 0.0245 -
0.5342 3800 0.0224 -
0.5413 3850 0.0226 -
0.5483 3900 0.0213 -
0.5553 3950 0.0207 -
0.5624 4000 0.0177 -
0.5694 4050 0.023 -
0.5764 4100 0.0203 -
0.5834 4150 0.0194 -
0.5905 4200 0.0183 -
0.5975 4250 0.0182 -
0.6045 4300 0.0164 -
0.6116 4350 0.0159 -
0.6186 4400 0.0147 -
0.6256 4450 0.0135 -
0.6326 4500 0.0131 -
0.6397 4550 0.0111 -
0.6467 4600 0.012 -
0.6537 4650 0.011 -
0.6608 4700 0.0119 -
0.6678 4750 0.0113 -
0.6748 4800 0.0107 -
0.6819 4850 0.0092 -
0.6889 4900 0.0097 -
0.6959 4950 0.0087 -
0.7029 5000 0.0082 -
0.7100 5050 0.01 -
0.7170 5100 0.0094 -
0.7240 5150 0.0081 -
0.7311 5200 0.0073 -
0.7381 5250 0.0069 -
0.7451 5300 0.0067 -
0.7521 5350 0.0072 -
0.7592 5400 0.0067 -
0.7662 5450 0.008 -
0.7732 5500 0.0072 -
0.7803 5550 0.0059 -
0.7873 5600 0.0059 -
0.7943 5650 0.0062 -
0.8013 5700 0.0068 -
0.8084 5750 0.0066 -
0.8154 5800 0.007 -
0.8224 5850 0.0057 -
0.8295 5900 0.0063 -
0.8365 5950 0.0062 -
0.8435 6000 0.0057 -
0.8506 6050 0.0051 -
0.8576 6100 0.0055 -
0.8646 6150 0.0046 -
0.8716 6200 0.0066 -
0.8787 6250 0.0054 -
0.8857 6300 0.004 -
0.8927 6350 0.0047 -
0.8998 6400 0.0044 -
0.9068 6450 0.004 -
0.9138 6500 0.005 -
0.9208 6550 0.0043 -
0.9279 6600 0.0039 -
0.9349 6650 0.0053 -
0.9419 6700 0.0039 -
0.9490 6750 0.0043 -
0.9560 6800 0.0045 -
0.9630 6850 0.0044 -
0.9701 6900 0.0041 -
0.9771 6950 0.0033 -
0.9841 7000 0.0031 -
0.9911 7050 0.0031 -
0.9982 7100 0.0039 -
1.0052 7150 0.0041 -
1.0122 7200 0.0026 -
1.0193 7250 0.0035 -
1.0263 7300 0.0032 -
1.0333 7350 0.0033 -
1.0403 7400 0.003 -
1.0474 7450 0.0025 -
1.0544 7500 0.0028 -
1.0614 7550 0.0027 -
1.0685 7600 0.0024 -
1.0755 7650 0.0026 -
1.0825 7700 0.0024 -
1.0896 7750 0.003 -
1.0966 7800 0.0024 -
1.1036 7850 0.0028 -
1.1106 7900 0.0023 -
1.1177 7950 0.0021 -
1.1247 8000 0.0028 -
1.1317 8050 0.0017 -
1.1388 8100 0.0022 -
1.1458 8150 0.0028 -
1.1528 8200 0.0019 -
1.1598 8250 0.002 -
1.1669 8300 0.0025 -
1.1739 8350 0.0027 -
1.1809 8400 0.0025 -
1.1880 8450 0.0018 -
1.1950 8500 0.0024 -
1.2020 8550 0.002 -
1.2091 8600 0.002 -
1.2161 8650 0.0019 -
1.2231 8700 0.0016 -
1.2301 8750 0.0018 -
1.2372 8800 0.0015 -
1.2442 8850 0.0017 -
1.2512 8900 0.0017 -
1.2583 8950 0.0019 -
1.2653 9000 0.0014 -
1.2723 9050 0.0016 -
1.2793 9100 0.0014 -
1.2864 9150 0.0017 -
1.2934 9200 0.0015 -
1.3004 9250 0.001 -
1.3075 9300 0.0017 -
1.3145 9350 0.0014 -
1.3215 9400 0.0013 -
1.3286 9450 0.0012 -
1.3356 9500 0.001 -
1.3426 9550 0.0014 -
1.3496 9600 0.0011 -
1.3567 9650 0.0012 -
1.3637 9700 0.0012 -
1.3707 9750 0.002 -
1.3778 9800 0.0012 -
1.3848 9850 0.0012 -
1.3918 9900 0.0016 -
1.3988 9950 0.0014 -
1.4059 10000 0.0011 -
1.4129 10050 0.0012 -
1.4199 10100 0.0013 -
1.4270 10150 0.0011 -
1.4340 10200 0.001 -
1.4410 10250 0.0016 -
1.4481 10300 0.0012 -
1.4551 10350 0.0012 -
1.4621 10400 0.0015 -
1.4691 10450 0.0014 -
1.4762 10500 0.0017 -
1.4832 10550 0.0019 -
1.4902 10600 0.0013 -
1.4973 10650 0.0012 -
1.5043 10700 0.0015 -
1.5113 10750 0.0012 -
1.5183 10800 0.0011 -
1.5254 10850 0.0015 -
1.5324 10900 0.001 -
1.5394 10950 0.001 -
1.5465 11000 0.0008 -
1.5535 11050 0.0014 -
1.5605 11100 0.0011 -
1.5676 11150 0.0014 -
1.5746 11200 0.0013 -
1.5816 11250 0.0008 -
1.5886 11300 0.001 -
1.5957 11350 0.0009 -
1.6027 11400 0.0014 -
1.6097 11450 0.0008 -
1.6168 11500 0.0014 -
1.6238 11550 0.0011 -
1.6308 11600 0.0015 -
1.6378 11650 0.0011 -
1.6449 11700 0.0006 -
1.6519 11750 0.0015 -
1.6589 11800 0.0015 -
1.6660 11850 0.0012 -
1.6730 11900 0.0008 -
1.6800 11950 0.0007 -
1.6871 12000 0.0011 -
1.6941 12050 0.0008 -
1.7011 12100 0.001 -
1.7081 12150 0.0008 -
1.7152 12200 0.0008 -
1.7222 12250 0.0011 -
1.7292 12300 0.0018 -
1.7363 12350 0.0007 -
1.7433 12400 0.0011 -
1.7503 12450 0.0006 -
1.7573 12500 0.0007 -
1.7644 12550 0.0012 -
1.7714 12600 0.001 -
1.7784 12650 0.0009 -
1.7855 12700 0.0009 -
1.7925 12750 0.0008 -
1.7995 12800 0.0009 -
1.8066 12850 0.0009 -
1.8136 12900 0.0013 -
1.8206 12950 0.0009 -
1.8276 13000 0.0011 -
1.8347 13050 0.0009 -
1.8417 13100 0.0007 -
1.8487 13150 0.0009 -
1.8558 13200 0.0008 -
1.8628 13250 0.0008 -
1.8698 13300 0.0006 -
1.8768 13350 0.0008 -
1.8839 13400 0.0007 -
1.8909 13450 0.0009 -
1.8979 13500 0.0005 -
1.9050 13550 0.0004 -
1.9120 13600 0.0014 -
1.9190 13650 0.0009 -
1.9261 13700 0.0009 -
1.9331 13750 0.001 -
1.9401 13800 0.0007 -
1.9471 13850 0.0011 -
1.9542 13900 0.0009 -
1.9612 13950 0.0005 -
1.9682 14000 0.0004 -
1.9753 14050 0.0007 -
1.9823 14100 0.0007 -
1.9893 14150 0.0006 -
1.9963 14200 0.0008 -
2.0034 14250 0.0005 -
2.0104 14300 0.0009 -
2.0174 14350 0.0004 -
2.0245 14400 0.0007 -
2.0315 14450 0.0007 -
2.0385 14500 0.0005 -
2.0456 14550 0.0007 -
2.0526 14600 0.0005 -
2.0596 14650 0.0004 -
2.0666 14700 0.0006 -
2.0737 14750 0.0007 -
2.0807 14800 0.0008 -
2.0877 14850 0.0009 -
2.0948 14900 0.0003 -
2.1018 14950 0.0006 -
2.1088 15000 0.0006 -
2.1158 15050 0.0007 -
2.1229 15100 0.0007 -
2.1299 15150 0.001 -
2.1369 15200 0.0006 -
2.1440 15250 0.0005 -
2.1510 15300 0.0008 -
2.1580 15350 0.0004 -
2.1650 15400 0.0006 -
2.1721 15450 0.0006 -
2.1791 15500 0.0008 -
2.1861 15550 0.0003 -
2.1932 15600 0.001 -
2.2002 15650 0.0005 -
2.2072 15700 0.0006 -
2.2143 15750 0.0005 -
2.2213 15800 0.0006 -
2.2283 15850 0.0005 -
2.2353 15900 0.0006 -
2.2424 15950 0.0003 -
2.2494 16000 0.0005 -
2.2564 16050 0.0005 -
2.2635 16100 0.0006 -
2.2705 16150 0.0006 -
2.2775 16200 0.0006 -
2.2845 16250 0.0005 -
2.2916 16300 0.0006 -
2.2986 16350 0.0004 -
2.3056 16400 0.0006 -
2.3127 16450 0.0004 -
2.3197 16500 0.0004 -
2.3267 16550 0.0007 -
2.3338 16600 0.0006 -
2.3408 16650 0.0004 -
2.3478 16700 0.0007 -
2.3548 16750 0.0009 -
2.3619 16800 0.0005 -
2.3689 16850 0.0005 -
2.3759 16900 0.0007 -
2.3830 16950 0.0005 -
2.3900 17000 0.0005 -
2.3970 17050 0.0008 -
2.4040 17100 0.0005 -
2.4111 17150 0.0004 -
2.4181 17200 0.0004 -
2.4251 17250 0.0006 -
2.4322 17300 0.0007 -
2.4392 17350 0.0005 -
2.4462 17400 0.0004 -
2.4533 17450 0.0003 -
2.4603 17500 0.0004 -
2.4673 17550 0.0008 -
2.4743 17600 0.0003 -
2.4814 17650 0.0006 -
2.4884 17700 0.0003 -
2.4954 17750 0.0004 -
2.5025 17800 0.0005 -
2.5095 17850 0.0004 -
2.5165 17900 0.0004 -
2.5235 17950 0.0003 -
2.5306 18000 0.0005 -
2.5376 18050 0.0005 -
2.5446 18100 0.0005 -
2.5517 18150 0.0004 -
2.5587 18200 0.0003 -
2.5657 18250 0.0007 -
2.5728 18300 0.0006 -
2.5798 18350 0.0004 -
2.5868 18400 0.0005 -
2.5938 18450 0.0006 -
2.6009 18500 0.0006 -
2.6079 18550 0.0006 -
2.6149 18600 0.0003 -
2.6220 18650 0.0003 -
2.6290 18700 0.0003 -
2.6360 18750 0.0004 -
2.6430 18800 0.0003 -
2.6501 18850 0.0005 -
2.6571 18900 0.0006 -
2.6641 18950 0.0003 -
2.6712 19000 0.0007 -
2.6782 19050 0.0002 -
2.6852 19100 0.0003 -
2.6923 19150 0.0004 -
2.6993 19200 0.0005 -
2.7063 19250 0.0004 -
2.7133 19300 0.0004 -
2.7204 19350 0.0006 -
2.7274 19400 0.0003 -
2.7344 19450 0.0002 -
2.7415 19500 0.0003 -
2.7485 19550 0.0004 -
2.7555 19600 0.0005 -
2.7625 19650 0.0004 -
2.7696 19700 0.0007 -
2.7766 19750 0.0003 -
2.7836 19800 0.0004 -
2.7907 19850 0.0006 -
2.7977 19900 0.0005 -
2.8047 19950 0.0004 -
2.8118 20000 0.0003 -
2.8188 20050 0.0007 -
2.8258 20100 0.0005 -
2.8328 20150 0.0006 -
2.8399 20200 0.0003 -
2.8469 20250 0.0008 -
2.8539 20300 0.0004 -
2.8610 20350 0.0007 -
2.8680 20400 0.0003 -
2.8750 20450 0.0005 -
2.8820 20500 0.0006 -
2.8891 20550 0.0002 -
2.8961 20600 0.0003 -
2.9031 20650 0.0004 -
2.9102 20700 0.0003 -
2.9172 20750 0.0004 -
2.9242 20800 0.0002 -
2.9313 20850 0.0003 -
2.9383 20900 0.0004 -
2.9453 20950 0.0005 -
2.9523 21000 0.0003 -
2.9594 21050 0.0005 -
2.9664 21100 0.0006 -
2.9734 21150 0.0003 -
2.9805 21200 0.0005 -
2.9875 21250 0.0003 -
2.9945 21300 0.0003 -

Framework Versions

  • Python: 3.13.7
  • SetFit: 1.1.3
  • Sentence Transformers: 5.1.1
  • Transformers: 4.57.0
  • PyTorch: 2.8.0+cu129
  • Datasets: 4.2.0
  • Tokenizers: 0.22.1

Citation

BibTeX 

@article{https://doi.org/10.48550/arxiv.2209.11055,
    doi = {10.48550/ARXIV.2209.11055},
    url = {https://arxiv.org/abs/2209.11055},
    author = {Tunstall, Lewis and Reimers, Nils and Jo, Unso Eun Seo and Bates, Luke and Korat, Daniel and Wasserblat, Moshe and Pereg, Oren},
    keywords = {Computation and Language (cs.CL), FOS: Computer and information sciences, FOS: Computer and information sciences},
    title = {Efficient Few-Shot Learning Without Prompts},
    publisher = {arXiv},
    year = {2022},
    copyright = {Creative Commons Attribution 4.0 International}
}