opt-babylm2-rewritten-clean-spacy-earlystop_hierarchical_211_size-origin_both-bpe_seed-42_1e-3

This model was trained from scratch on the kanishka/babylm2-rewritten-clean-spacy_hierarchical-adj_211_size-origin_both-ablation dataset. It achieves the following results on the evaluation set:

• Loss: 2.6851
• Accuracy: 0.4783

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 0.001
• train_batch_size: 32
• eval_batch_size: 64
• seed: 42
• gradient_accumulation_steps: 8
• total_train_batch_size: 256
• optimizer: Use OptimizerNames.ADAMW_TORCH with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
• lr_scheduler_type: linear
• lr_scheduler_warmup_steps: 32000
• num_epochs: 20.0
• mixed_precision_training: Native AMP

Training results

Training Loss	Epoch	Step	Validation Loss	Accuracy
4.094	0.9997	2255	3.8037	0.3619
3.443	1.9997	4510	3.2928	0.4105
3.1257	2.9997	6765	3.0869	0.4308
2.9222	3.9997	9020	2.9820	0.4411
2.8442	4.9997	11275	2.9212	0.4474
2.7882	5.9997	13530	2.8832	0.4514
2.7435	6.9997	15785	2.8560	0.4547
2.7124	7.9997	18040	2.8370	0.4567
2.6895	8.9997	20295	2.8221	0.4585
2.6692	9.9997	22550	2.8129	0.4598
2.6489	10.9997	24805	2.8064	0.4605
2.6509	11.9997	27060	2.7973	0.4615
2.6391	12.9997	29315	2.7914	0.4621
2.6299	13.9997	31570	2.7891	0.4626
2.6116	14.9997	33825	2.7651	0.4652
2.5694	15.9997	36080	2.7420	0.4683
2.5208	16.9997	38335	2.7206	0.4715
2.4644	17.9997	40590	2.7018	0.4742
2.3999	18.9997	42845	2.6869	0.4769
2.3272	19.9997	45100	2.6851	0.4783

Framework versions

• Transformers 4.48.0
• Pytorch 2.6.0+cu124
• Datasets 3.2.0
• Tokenizers 0.21.4