Hulyyy/req-quality-setfit-128

SetFit

This is a SetFit model that can be used for Text Classification. A WeightedBinaryRelevanceHead instance is used for classification.

The model has been trained using an efficient few-shot learning technique that involves:

1. Fine-tuning a Sentence Transformer with contrastive learning.
2. Training a classification head with features from the fine-tuned Sentence Transformer.

Model Details

Model Description

• Model Type: SetFit
• Classification head: a WeightedBinaryRelevanceHead instance
• Maximum Sequence Length: 256 tokens

Model Sources

• Repository: SetFit on GitHub
• Paper: Efficient Few-Shot Learning Without Prompts
• Blogpost: SetFit: Efficient Few-Shot Learning Without Prompts

Evaluation

Metrics

Label	Micro_F1	Macro_F1	Hamming_Accuracy	Subset_Accuracy
all	0.5631	0.5742	0.8889	0.5638

Uses

Direct Use for Inference

First install the SetFit library:

pip install setfit

Then you can load this model and run inference.

from setfit import SetFitModel

# Download from the 🤗 Hub
model = SetFitModel.from_pretrained("Hulyyy/req-quality-setfit-128")
# Run inference
preds = model("The product shall store new conference rooms.")

Training Details

Training Set Metrics

Training set	Min	Median	Max
Word count	2	19.3840	32

Training Hyperparameters

• batch_size: (64, 64)
• num_epochs: (3, 3)
• max_steps: -1
• sampling_strategy: oversampling
• num_iterations: 200
• body_learning_rate: (1e-05, 1e-05)
• head_learning_rate: 1e-05
• loss: CosineSimilarityLoss
• distance_metric: cosine_distance
• margin: 0.25
• end_to_end: False
• use_amp: True
• warmup_proportion: 0.1
• l2_weight: 0.01
• seed: 42
• eval_max_steps: -1
• load_best_model_at_end: False

Training Results

Epoch	Step	Training Loss	Validation Loss
0.0001	1	0.3722	-
0.0070	50	0.3592	-
0.0141	100	0.3552	-
0.0211	150	0.3271	-
0.0281	200	0.3016	-
0.0351	250	0.2783	-
0.0422	300	0.2642	-
0.0492	350	0.2587	-
0.0562	400	0.2568	-
0.0633	450	0.257	-
0.0703	500	0.2498	-
0.0773	550	0.2444	-
0.0844	600	0.2437	-
0.0914	650	0.2399	-
0.0984	700	0.2371	-
0.1054	750	0.2302	-
0.1125	800	0.2291	-
0.1195	850	0.2233	-
0.1265	900	0.2192	-
0.1336	950	0.2229	-
0.1406	1000	0.218	-
0.1476	1050	0.2096	-
0.1546	1100	0.2107	-
0.1617	1150	0.202	-
0.1687	1200	0.2009	-
0.1757	1250	0.1975	-
0.1828	1300	0.1969	-
0.1898	1350	0.1955	-
0.1968	1400	0.1871	-
0.2039	1450	0.1837	-
0.2109	1500	0.1732	-
0.2179	1550	0.1702	-
0.2249	1600	0.1687	-
0.2320	1650	0.16	-
0.2390	1700	0.1551	-
0.2460	1750	0.1495	-
0.2531	1800	0.1386	-
0.2601	1850	0.1339	-
0.2671	1900	0.1263	-
0.2741	1950	0.1232	-
0.2812	2000	0.1209	-
0.2882	2050	0.112	-
0.2952	2100	0.0995	-
0.3023	2150	0.0932	-
0.3093	2200	0.0869	-
0.3163	2250	0.076	-
0.3234	2300	0.0656	-
0.3304	2350	0.067	-
0.3374	2400	0.0602	-
0.3444	2450	0.0561	-
0.3515	2500	0.0532	-
0.3585	2550	0.0465	-
0.3655	2600	0.0443	-
0.3726	2650	0.0443	-
0.3796	2700	0.0435	-
0.3866	2750	0.0392	-
0.3936	2800	0.0417	-
0.4007	2850	0.0354	-
0.4077	2900	0.0379	-
0.4147	2950	0.037	-
0.4218	3000	0.0366	-
0.4288	3050	0.0328	-
0.4358	3100	0.032	-
0.4429	3150	0.027	-
0.4499	3200	0.0329	-
0.4569	3250	0.026	-
0.4639	3300	0.0284	-
0.4710	3350	0.0274	-
0.4780	3400	0.0264	-
0.4850	3450	0.0239	-
0.4921	3500	0.0256	-
0.4991	3550	0.0256	-
0.5061	3600	0.024	-
0.5131	3650	0.0223	-
0.5202	3700	0.0225	-
0.5272	3750	0.0245	-
0.5342	3800	0.0224	-
0.5413	3850	0.0226	-
0.5483	3900	0.0213	-
0.5553	3950	0.0207	-
0.5624	4000	0.0177	-
0.5694	4050	0.023	-
0.5764	4100	0.0203	-
0.5834	4150	0.0194	-
0.5905	4200	0.0183	-
0.5975	4250	0.0182	-
0.6045	4300	0.0164	-
0.6116	4350	0.0159	-
0.6186	4400	0.0147	-
0.6256	4450	0.0135	-
0.6326	4500	0.0131	-
0.6397	4550	0.0111	-
0.6467	4600	0.012	-
0.6537	4650	0.011	-
0.6608	4700	0.0119	-
0.6678	4750	0.0113	-
0.6748	4800	0.0107	-
0.6819	4850	0.0092	-
0.6889	4900	0.0097	-
0.6959	4950	0.0087	-
0.7029	5000	0.0082	-
0.7100	5050	0.01	-
0.7170	5100	0.0094	-
0.7240	5150	0.0081	-
0.7311	5200	0.0073	-
0.7381	5250	0.0069	-
0.7451	5300	0.0067	-
0.7521	5350	0.0072	-
0.7592	5400	0.0067	-
0.7662	5450	0.008	-
0.7732	5500	0.0072	-
0.7803	5550	0.0059	-
0.7873	5600	0.0059	-
0.7943	5650	0.0062	-
0.8013	5700	0.0068	-
0.8084	5750	0.0066	-
0.8154	5800	0.007	-
0.8224	5850	0.0057	-
0.8295	5900	0.0063	-
0.8365	5950	0.0062	-
0.8435	6000	0.0057	-
0.8506	6050	0.0051	-
0.8576	6100	0.0055	-
0.8646	6150	0.0046	-
0.8716	6200	0.0066	-
0.8787	6250	0.0054	-
0.8857	6300	0.004	-
0.8927	6350	0.0047	-
0.8998	6400	0.0044	-
0.9068	6450	0.004	-
0.9138	6500	0.005	-
0.9208	6550	0.0043	-
0.9279	6600	0.0039	-
0.9349	6650	0.0053	-
0.9419	6700	0.0039	-
0.9490	6750	0.0043	-
0.9560	6800	0.0045	-
0.9630	6850	0.0044	-
0.9701	6900	0.0041	-
0.9771	6950	0.0033	-
0.9841	7000	0.0031	-
0.9911	7050	0.0031	-
0.9982	7100	0.0039	-
1.0052	7150	0.0041	-
1.0122	7200	0.0026	-
1.0193	7250	0.0035	-
1.0263	7300	0.0032	-
1.0333	7350	0.0033	-
1.0403	7400	0.003	-
1.0474	7450	0.0025	-
1.0544	7500	0.0028	-
1.0614	7550	0.0027	-
1.0685	7600	0.0024	-
1.0755	7650	0.0026	-
1.0825	7700	0.0024	-
1.0896	7750	0.003	-
1.0966	7800	0.0024	-
1.1036	7850	0.0028	-
1.1106	7900	0.0023	-
1.1177	7950	0.0021	-
1.1247	8000	0.0028	-
1.1317	8050	0.0017	-
1.1388	8100	0.0022	-
1.1458	8150	0.0028	-
1.1528	8200	0.0019	-
1.1598	8250	0.002	-
1.1669	8300	0.0025	-
1.1739	8350	0.0027	-
1.1809	8400	0.0025	-
1.1880	8450	0.0018	-
1.1950	8500	0.0024	-
1.2020	8550	0.002	-
1.2091	8600	0.002	-
1.2161	8650	0.0019	-
1.2231	8700	0.0016	-
1.2301	8750	0.0018	-
1.2372	8800	0.0015	-
1.2442	8850	0.0017	-
1.2512	8900	0.0017	-
1.2583	8950	0.0019	-
1.2653	9000	0.0014	-
1.2723	9050	0.0016	-
1.2793	9100	0.0014	-
1.2864	9150	0.0017	-
1.2934	9200	0.0015	-
1.3004	9250	0.001	-
1.3075	9300	0.0017	-
1.3145	9350	0.0014	-
1.3215	9400	0.0013	-
1.3286	9450	0.0012	-
1.3356	9500	0.001	-
1.3426	9550	0.0014	-
1.3496	9600	0.0011	-
1.3567	9650	0.0012	-
1.3637	9700	0.0012	-
1.3707	9750	0.002	-
1.3778	9800	0.0012	-
1.3848	9850	0.0012	-
1.3918	9900	0.0016	-
1.3988	9950	0.0014	-
1.4059	10000	0.0011	-
1.4129	10050	0.0012	-
1.4199	10100	0.0013	-
1.4270	10150	0.0011	-
1.4340	10200	0.001	-
1.4410	10250	0.0016	-
1.4481	10300	0.0012	-
1.4551	10350	0.0012	-
1.4621	10400	0.0015	-
1.4691	10450	0.0014	-
1.4762	10500	0.0017	-
1.4832	10550	0.0019	-
1.4902	10600	0.0013	-
1.4973	10650	0.0012	-
1.5043	10700	0.0015	-
1.5113	10750	0.0012	-
1.5183	10800	0.0011	-
1.5254	10850	0.0015	-
1.5324	10900	0.001	-
1.5394	10950	0.001	-
1.5465	11000	0.0008	-
1.5535	11050	0.0014	-
1.5605	11100	0.0011	-
1.5676	11150	0.0014	-
1.5746	11200	0.0013	-
1.5816	11250	0.0008	-
1.5886	11300	0.001	-
1.5957	11350	0.0009	-
1.6027	11400	0.0014	-
1.6097	11450	0.0008	-
1.6168	11500	0.0014	-
1.6238	11550	0.0011	-
1.6308	11600	0.0015	-
1.6378	11650	0.0011	-
1.6449	11700	0.0006	-
1.6519	11750	0.0015	-
1.6589	11800	0.0015	-
1.6660	11850	0.0012	-
1.6730	11900	0.0008	-
1.6800	11950	0.0007	-
1.6871	12000	0.0011	-
1.6941	12050	0.0008	-
1.7011	12100	0.001	-
1.7081	12150	0.0008	-
1.7152	12200	0.0008	-
1.7222	12250	0.0011	-
1.7292	12300	0.0018	-
1.7363	12350	0.0007	-
1.7433	12400	0.0011	-
1.7503	12450	0.0006	-
1.7573	12500	0.0007	-
1.7644	12550	0.0012	-
1.7714	12600	0.001	-
1.7784	12650	0.0009	-
1.7855	12700	0.0009	-
1.7925	12750	0.0008	-
1.7995	12800	0.0009	-
1.8066	12850	0.0009	-
1.8136	12900	0.0013	-
1.8206	12950	0.0009	-
1.8276	13000	0.0011	-
1.8347	13050	0.0009	-
1.8417	13100	0.0007	-
1.8487	13150	0.0009	-
1.8558	13200	0.0008	-
1.8628	13250	0.0008	-
1.8698	13300	0.0006	-
1.8768	13350	0.0008	-
1.8839	13400	0.0007	-
1.8909	13450	0.0009	-
1.8979	13500	0.0005	-
1.9050	13550	0.0004	-
1.9120	13600	0.0014	-
1.9190	13650	0.0009	-
1.9261	13700	0.0009	-
1.9331	13750	0.001	-
1.9401	13800	0.0007	-
1.9471	13850	0.0011	-
1.9542	13900	0.0009	-
1.9612	13950	0.0005	-
1.9682	14000	0.0004	-
1.9753	14050	0.0007	-
1.9823	14100	0.0007	-
1.9893	14150	0.0006	-
1.9963	14200	0.0008	-
2.0034	14250	0.0005	-
2.0104	14300	0.0009	-
2.0174	14350	0.0004	-
2.0245	14400	0.0007	-
2.0315	14450	0.0007	-
2.0385	14500	0.0005	-
2.0456	14550	0.0007	-
2.0526	14600	0.0005	-
2.0596	14650	0.0004	-
2.0666	14700	0.0006	-
2.0737	14750	0.0007	-
2.0807	14800	0.0008	-
2.0877	14850	0.0009	-
2.0948	14900	0.0003	-
2.1018	14950	0.0006	-
2.1088	15000	0.0006	-
2.1158	15050	0.0007	-
2.1229	15100	0.0007	-
2.1299	15150	0.001	-
2.1369	15200	0.0006	-
2.1440	15250	0.0005	-
2.1510	15300	0.0008	-
2.1580	15350	0.0004	-
2.1650	15400	0.0006	-
2.1721	15450	0.0006	-
2.1791	15500	0.0008	-
2.1861	15550	0.0003	-
2.1932	15600	0.001	-
2.2002	15650	0.0005	-
2.2072	15700	0.0006	-
2.2143	15750	0.0005	-
2.2213	15800	0.0006	-
2.2283	15850	0.0005	-
2.2353	15900	0.0006	-
2.2424	15950	0.0003	-
2.2494	16000	0.0005	-
2.2564	16050	0.0005	-
2.2635	16100	0.0006	-
2.2705	16150	0.0006	-
2.2775	16200	0.0006	-
2.2845	16250	0.0005	-
2.2916	16300	0.0006	-
2.2986	16350	0.0004	-
2.3056	16400	0.0006	-
2.3127	16450	0.0004	-
2.3197	16500	0.0004	-
2.3267	16550	0.0007	-
2.3338	16600	0.0006	-
2.3408	16650	0.0004	-
2.3478	16700	0.0007	-
2.3548	16750	0.0009	-
2.3619	16800	0.0005	-
2.3689	16850	0.0005	-
2.3759	16900	0.0007	-
2.3830	16950	0.0005	-
2.3900	17000	0.0005	-
2.3970	17050	0.0008	-
2.4040	17100	0.0005	-
2.4111	17150	0.0004	-
2.4181	17200	0.0004	-
2.4251	17250	0.0006	-
2.4322	17300	0.0007	-
2.4392	17350	0.0005	-
2.4462	17400	0.0004	-
2.4533	17450	0.0003	-
2.4603	17500	0.0004	-
2.4673	17550	0.0008	-
2.4743	17600	0.0003	-
2.4814	17650	0.0006	-
2.4884	17700	0.0003	-
2.4954	17750	0.0004	-
2.5025	17800	0.0005	-
2.5095	17850	0.0004	-
2.5165	17900	0.0004	-
2.5235	17950	0.0003	-
2.5306	18000	0.0005	-
2.5376	18050	0.0005	-
2.5446	18100	0.0005	-
2.5517	18150	0.0004	-
2.5587	18200	0.0003	-
2.5657	18250	0.0007	-
2.5728	18300	0.0006	-
2.5798	18350	0.0004	-
2.5868	18400	0.0005	-
2.5938	18450	0.0006	-
2.6009	18500	0.0006	-
2.6079	18550	0.0006	-
2.6149	18600	0.0003	-
2.6220	18650	0.0003	-
2.6290	18700	0.0003	-
2.6360	18750	0.0004	-
2.6430	18800	0.0003	-
2.6501	18850	0.0005	-
2.6571	18900	0.0006	-
2.6641	18950	0.0003	-
2.6712	19000	0.0007	-
2.6782	19050	0.0002	-
2.6852	19100	0.0003	-
2.6923	19150	0.0004	-
2.6993	19200	0.0005	-
2.7063	19250	0.0004	-
2.7133	19300	0.0004	-
2.7204	19350	0.0006	-
2.7274	19400	0.0003	-
2.7344	19450	0.0002	-
2.7415	19500	0.0003	-
2.7485	19550	0.0004	-
2.7555	19600	0.0005	-
2.7625	19650	0.0004	-
2.7696	19700	0.0007	-
2.7766	19750	0.0003	-
2.7836	19800	0.0004	-
2.7907	19850	0.0006	-
2.7977	19900	0.0005	-
2.8047	19950	0.0004	-
2.8118	20000	0.0003	-
2.8188	20050	0.0007	-
2.8258	20100	0.0005	-
2.8328	20150	0.0006	-
2.8399	20200	0.0003	-
2.8469	20250	0.0008	-
2.8539	20300	0.0004	-
2.8610	20350	0.0007	-
2.8680	20400	0.0003	-
2.8750	20450	0.0005	-
2.8820	20500	0.0006	-
2.8891	20550	0.0002	-
2.8961	20600	0.0003	-
2.9031	20650	0.0004	-
2.9102	20700	0.0003	-
2.9172	20750	0.0004	-
2.9242	20800	0.0002	-
2.9313	20850	0.0003	-
2.9383	20900	0.0004	-
2.9453	20950	0.0005	-
2.9523	21000	0.0003	-
2.9594	21050	0.0005	-
2.9664	21100	0.0006	-
2.9734	21150	0.0003	-
2.9805	21200	0.0005	-
2.9875	21250	0.0003	-
2.9945	21300	0.0003	-

Framework Versions

• Python: 3.13.7
• SetFit: 1.1.3
• Sentence Transformers: 5.1.1
• Transformers: 4.57.0
• PyTorch: 2.8.0+cu129
• Datasets: 4.2.0
• Tokenizers: 0.22.1

Citation

BibTeX

@article{https://doi.org/10.48550/arxiv.2209.11055,
    doi = {10.48550/ARXIV.2209.11055},
    url = {https://arxiv.org/abs/2209.11055},
    author = {Tunstall, Lewis and Reimers, Nils and Jo, Unso Eun Seo and Bates, Luke and Korat, Daniel and Wasserblat, Moshe and Pereg, Oren},
    keywords = {Computation and Language (cs.CL), FOS: Computer and information sciences, FOS: Computer and information sciences},
    title = {Efficient Few-Shot Learning Without Prompts},
    publisher = {arXiv},
    year = {2022},
    copyright = {Creative Commons Attribution 4.0 International}
}