--- library_name: peft license: llama3 base_model: meta-llama/Meta-Llama-3-8B-Instruct tags: - llama-factory - prompt-tuning - generated_from_trainer model-index: - name: train_cola_1744902672 results: [] --- # train_cola_1744902672 This model is a fine-tuned version of [meta-llama/Meta-Llama-3-8B-Instruct](https://huggingface.co/meta-llama/Meta-Llama-3-8B-Instruct) on the cola dataset. It achieves the following results on the evaluation set: - Loss: 0.2466 - Num Input Tokens Seen: 30508240 ## Model description More information needed ## Intended uses & limitations More information needed ## Training and evaluation data More information needed ## Training procedure ### Training hyperparameters The following hyperparameters were used during training: - learning_rate: 0.3 - train_batch_size: 4 - eval_batch_size: 4 - seed: 123 - gradient_accumulation_steps: 4 - total_train_batch_size: 16 - optimizer: Use adamw_torch with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments - lr_scheduler_type: cosine - training_steps: 40000 ### Training results | Training Loss | Epoch | Step | Validation Loss | Input Tokens Seen | |:-------------:|:-------:|:-----:|:---------------:|:-----------------:| | 0.3225 | 0.4158 | 200 | 0.3066 | 153120 | | 0.2547 | 0.8316 | 400 | 0.2711 | 305504 | | 0.4244 | 1.2474 | 600 | 0.2912 | 458648 | | 0.36 | 1.6632 | 800 | 0.2836 | 610680 | | 0.3203 | 2.0790 | 1000 | 0.2618 | 763880 | | 0.2574 | 2.4948 | 1200 | 0.2869 | 916648 | | 0.2787 | 2.9106 | 1400 | 0.2540 | 1068552 | | 0.2469 | 3.3264 | 1600 | 0.2597 | 1220928 | | 0.2798 | 3.7422 | 1800 | 0.2820 | 1373952 | | 0.2612 | 4.1580 | 2000 | 0.2518 | 1526312 | | 0.2863 | 4.5738 | 2200 | 0.2525 | 1678248 | | 0.3352 | 4.9896 | 2400 | 0.2801 | 1831112 | | 0.2685 | 5.4054 | 2600 | 0.2652 | 1983296 | | 0.3158 | 5.8212 | 2800 | 0.2623 | 2135968 | | 0.2789 | 6.2370 | 3000 | 0.2499 | 2289200 | | 0.2968 | 6.6528 | 3200 | 0.2987 | 2441648 | | 0.2783 | 7.0686 | 3400 | 0.2497 | 2593344 | | 0.3116 | 7.4844 | 3600 | 0.3089 | 2745792 | | 0.2829 | 7.9002 | 3800 | 0.2540 | 2898816 | | 0.2801 | 8.3160 | 4000 | 0.2654 | 3050480 | | 0.2669 | 8.7318 | 4200 | 0.2513 | 3202864 | | 0.2617 | 9.1476 | 4400 | 0.2511 | 3355680 | | 0.2678 | 9.5634 | 4600 | 0.2624 | 3508192 | | 0.249 | 9.9792 | 4800 | 0.2519 | 3661568 | | 0.3018 | 10.3950 | 5000 | 0.2741 | 3813552 | | 0.2871 | 10.8108 | 5200 | 0.2504 | 3967024 | | 0.2634 | 11.2266 | 5400 | 0.2494 | 4120032 | | 0.2821 | 11.6424 | 5600 | 0.2752 | 4272608 | | 0.2883 | 12.0582 | 5800 | 0.2677 | 4424280 | | 0.271 | 12.4740 | 6000 | 0.2492 | 4575480 | | 0.2644 | 12.8898 | 6200 | 0.2496 | 4728792 | | 0.2439 | 13.3056 | 6400 | 0.2497 | 4880880 | | 0.2949 | 13.7214 | 6600 | 0.2553 | 5034608 | | 0.2525 | 14.1372 | 6800 | 0.2501 | 5186400 | | 0.297 | 14.5530 | 7000 | 0.2512 | 5339008 | | 0.2724 | 14.9688 | 7200 | 0.2489 | 5491424 | | 0.2623 | 15.3846 | 7400 | 0.2484 | 5644520 | | 0.2063 | 15.8004 | 7600 | 0.2708 | 5796744 | | 0.276 | 16.2162 | 7800 | 0.2498 | 5949536 | | 0.3006 | 16.6320 | 8000 | 0.2608 | 6102304 | | 0.2718 | 17.0478 | 8200 | 0.2479 | 6254288 | | 0.2789 | 17.4636 | 8400 | 0.2646 | 6407504 | | 0.2784 | 17.8794 | 8600 | 0.2550 | 6559760 | | 0.2703 | 18.2952 | 8800 | 0.2493 | 6711968 | | 0.2681 | 18.7110 | 9000 | 0.2509 | 6864736 | | 0.286 | 19.1268 | 9200 | 0.2611 | 7016944 | | 0.2682 | 19.5426 | 9400 | 0.2485 | 7169456 | | 0.2833 | 19.9584 | 9600 | 0.2534 | 7322736 | | 0.2695 | 20.3742 | 9800 | 0.2516 | 7474848 | | 0.2552 | 20.7900 | 10000 | 0.2489 | 7627360 | | 0.2899 | 21.2058 | 10200 | 0.2829 | 7779952 | | 0.2748 | 21.6216 | 10400 | 0.2522 | 7932848 | | 0.2799 | 22.0374 | 10600 | 0.2573 | 8085448 | | 0.2724 | 22.4532 | 10800 | 0.2622 | 8237768 | | 0.2279 | 22.8690 | 11000 | 0.2505 | 8390664 | | 0.266 | 23.2848 | 11200 | 0.2536 | 8543280 | | 0.2712 | 23.7006 | 11400 | 0.2582 | 8696432 | | 0.2664 | 24.1164 | 11600 | 0.2500 | 8849408 | | 0.2668 | 24.5322 | 11800 | 0.2539 | 9001408 | | 0.2822 | 24.9480 | 12000 | 0.2769 | 9153696 | | 0.2745 | 25.3638 | 12200 | 0.2504 | 9307088 | | 0.2536 | 25.7796 | 12400 | 0.2482 | 9459824 | | 0.2723 | 26.1954 | 12600 | 0.2495 | 9611704 | | 0.2621 | 26.6112 | 12800 | 0.2492 | 9764344 | | 0.2696 | 27.0270 | 13000 | 0.2564 | 9917064 | | 0.2466 | 27.4428 | 13200 | 0.2527 | 10068520 | | 0.283 | 27.8586 | 13400 | 0.2568 | 10221224 | | 0.2742 | 28.2744 | 13600 | 0.2484 | 10373912 | | 0.2789 | 28.6902 | 13800 | 0.2497 | 10526808 | | 0.273 | 29.1060 | 14000 | 0.2591 | 10678976 | | 0.2554 | 29.5218 | 14200 | 0.2481 | 10831520 | | 0.2605 | 29.9376 | 14400 | 0.2485 | 10984224 | | 0.2916 | 30.3534 | 14600 | 0.2487 | 11135896 | | 0.2517 | 30.7692 | 14800 | 0.2503 | 11288728 | | 0.2847 | 31.1850 | 15000 | 0.2474 | 11441040 | | 0.2556 | 31.6008 | 15200 | 0.2535 | 11593456 | | 0.2735 | 32.0166 | 15400 | 0.2535 | 11745744 | | 0.2402 | 32.4324 | 15600 | 0.2488 | 11898672 | | 0.2652 | 32.8482 | 15800 | 0.2505 | 12050992 | | 0.246 | 33.2640 | 16000 | 0.2476 | 12204352 | | 0.2549 | 33.6798 | 16200 | 0.2490 | 12356224 | | 0.2714 | 34.0956 | 16400 | 0.2524 | 12507960 | | 0.2736 | 34.5114 | 16600 | 0.2697 | 12660760 | | 0.2601 | 34.9272 | 16800 | 0.2487 | 12813272 | | 0.2638 | 35.3430 | 17000 | 0.2489 | 12965896 | | 0.2766 | 35.7588 | 17200 | 0.2524 | 13118824 | | 0.2599 | 36.1746 | 17400 | 0.2490 | 13271872 | | 0.2689 | 36.5904 | 17600 | 0.2603 | 13424128 | | 0.2689 | 37.0062 | 17800 | 0.2657 | 13576056 | | 0.2675 | 37.4220 | 18000 | 0.2472 | 13728696 | | 0.2572 | 37.8378 | 18200 | 0.2515 | 13881368 | | 0.2781 | 38.2536 | 18400 | 0.2623 | 14033616 | | 0.2692 | 38.6694 | 18600 | 0.2496 | 14185616 | | 0.2311 | 39.0852 | 18800 | 0.2480 | 14338720 | | 0.2502 | 39.5010 | 19000 | 0.2497 | 14490240 | | 0.2788 | 39.9168 | 19200 | 0.2483 | 14643072 | | 0.2716 | 40.3326 | 19400 | 0.2582 | 14795184 | | 0.2627 | 40.7484 | 19600 | 0.2483 | 14947312 | | 0.2659 | 41.1642 | 19800 | 0.2634 | 15100336 | | 0.306 | 41.5800 | 20000 | 0.2473 | 15252464 | | 0.2733 | 41.9958 | 20200 | 0.2680 | 15404912 | | 0.2541 | 42.4116 | 20400 | 0.2489 | 15557176 | | 0.2715 | 42.8274 | 20600 | 0.2510 | 15709912 | | 0.2508 | 43.2432 | 20800 | 0.2493 | 15862336 | | 0.2563 | 43.6590 | 21000 | 0.2475 | 16014304 | | 0.2613 | 44.0748 | 21200 | 0.2476 | 16166680 | | 0.2594 | 44.4906 | 21400 | 0.2499 | 16320408 | | 0.2615 | 44.9064 | 21600 | 0.2489 | 16472888 | | 0.2551 | 45.3222 | 21800 | 0.2519 | 16625808 | | 0.261 | 45.7380 | 22000 | 0.2485 | 16778288 | | 0.2597 | 46.1538 | 22200 | 0.2487 | 16931560 | | 0.2781 | 46.5696 | 22400 | 0.2591 | 17083880 | | 0.2659 | 46.9854 | 22600 | 0.2583 | 17235976 | | 0.2489 | 47.4012 | 22800 | 0.2478 | 17388152 | | 0.2575 | 47.8170 | 23000 | 0.2510 | 17540824 | | 0.2172 | 48.2328 | 23200 | 0.2509 | 17693912 | | 0.2799 | 48.6486 | 23400 | 0.2471 | 17846296 | | 0.2467 | 49.0644 | 23600 | 0.2501 | 17998760 | | 0.2768 | 49.4802 | 23800 | 0.2510 | 18152072 | | 0.2944 | 49.8960 | 24000 | 0.2493 | 18304072 | | 0.2994 | 50.3119 | 24200 | 0.2466 | 18455696 | | 0.2625 | 50.7277 | 24400 | 0.2490 | 18608976 | | 0.2745 | 51.1435 | 24600 | 0.2602 | 18760928 | | 0.2414 | 51.5593 | 24800 | 0.2472 | 18913856 | | 0.2668 | 51.9751 | 25000 | 0.2488 | 19066528 | | 0.2498 | 52.3909 | 25200 | 0.2511 | 19218616 | | 0.2658 | 52.8067 | 25400 | 0.2578 | 19370872 | | 0.2538 | 53.2225 | 25600 | 0.2522 | 19524232 | | 0.2282 | 53.6383 | 25800 | 0.2493 | 19676456 | | 0.2705 | 54.0541 | 26000 | 0.2469 | 19828504 | | 0.2621 | 54.4699 | 26200 | 0.2472 | 19980856 | | 0.263 | 54.8857 | 26400 | 0.2538 | 20133784 | | 0.273 | 55.3015 | 26600 | 0.2565 | 20286120 | | 0.2596 | 55.7173 | 26800 | 0.2568 | 20439016 | | 0.2786 | 56.1331 | 27000 | 0.2581 | 20591320 | | 0.2734 | 56.5489 | 27200 | 0.2475 | 20743736 | | 0.2701 | 56.9647 | 27400 | 0.2573 | 20896184 | | 0.2462 | 57.3805 | 27600 | 0.2511 | 21049160 | | 0.269 | 57.7963 | 27800 | 0.2510 | 21201640 | | 0.2667 | 58.2121 | 28000 | 0.2472 | 21354208 | | 0.2785 | 58.6279 | 28200 | 0.2480 | 21506752 | | 0.2669 | 59.0437 | 28400 | 0.2576 | 21659696 | | 0.2614 | 59.4595 | 28600 | 0.2479 | 21811600 | | 0.2765 | 59.8753 | 28800 | 0.2478 | 21964272 | | 0.2601 | 60.2911 | 29000 | 0.2511 | 22116648 | | 0.2744 | 60.7069 | 29200 | 0.2514 | 22269032 | | 0.2519 | 61.1227 | 29400 | 0.2478 | 22421944 | | 0.2741 | 61.5385 | 29600 | 0.2526 | 22574936 | | 0.2615 | 61.9543 | 29800 | 0.2472 | 22727064 | | 0.2514 | 62.3701 | 30000 | 0.2474 | 22880256 | | 0.2572 | 62.7859 | 30200 | 0.2502 | 23032800 | | 0.2698 | 63.2017 | 30400 | 0.2473 | 23184744 | | 0.2654 | 63.6175 | 30600 | 0.2473 | 23336904 | | 0.2354 | 64.0333 | 30800 | 0.2482 | 23489432 | | 0.2766 | 64.4491 | 31000 | 0.2471 | 23641496 | | 0.2808 | 64.8649 | 31200 | 0.2504 | 23794744 | | 0.2555 | 65.2807 | 31400 | 0.2505 | 23947688 | | 0.2635 | 65.6965 | 31600 | 0.2521 | 24099432 | | 0.2697 | 66.1123 | 31800 | 0.2474 | 24251200 | | 0.2765 | 66.5281 | 32000 | 0.2498 | 24404736 | | 0.2743 | 66.9439 | 32200 | 0.2487 | 24557120 | | 0.2775 | 67.3597 | 32400 | 0.2504 | 24709616 | | 0.2678 | 67.7755 | 32600 | 0.2499 | 24862224 | | 0.2658 | 68.1913 | 32800 | 0.2485 | 25015296 | | 0.2544 | 68.6071 | 33000 | 0.2480 | 25167744 | | 0.2748 | 69.0229 | 33200 | 0.2526 | 25321016 | | 0.2713 | 69.4387 | 33400 | 0.2497 | 25473368 | | 0.2548 | 69.8545 | 33600 | 0.2491 | 25626520 | | 0.2589 | 70.2703 | 33800 | 0.2492 | 25778248 | | 0.2534 | 70.6861 | 34000 | 0.2510 | 25930920 | | 0.2532 | 71.1019 | 34200 | 0.2491 | 26083456 | | 0.266 | 71.5177 | 34400 | 0.2504 | 26235552 | | 0.256 | 71.9335 | 34600 | 0.2535 | 26388832 | | 0.2666 | 72.3493 | 34800 | 0.2476 | 26541680 | | 0.2528 | 72.7651 | 35000 | 0.2502 | 26694832 | | 0.263 | 73.1809 | 35200 | 0.2483 | 26847168 | | 0.2583 | 73.5967 | 35400 | 0.2481 | 27000096 | | 0.277 | 74.0125 | 35600 | 0.2492 | 27151800 | | 0.2486 | 74.4283 | 35800 | 0.2479 | 27304152 | | 0.2598 | 74.8441 | 36000 | 0.2508 | 27456856 | | 0.2815 | 75.2599 | 36200 | 0.2513 | 27610376 | | 0.2609 | 75.6757 | 36400 | 0.2488 | 27762984 | | 0.2621 | 76.0915 | 36600 | 0.2487 | 27915504 | | 0.2488 | 76.5073 | 36800 | 0.2484 | 28068432 | | 0.2812 | 76.9231 | 37000 | 0.2496 | 28220720 | | 0.276 | 77.3389 | 37200 | 0.2491 | 28373600 | | 0.2792 | 77.7547 | 37400 | 0.2488 | 28526304 | | 0.2478 | 78.1705 | 37600 | 0.2498 | 28678672 | | 0.2691 | 78.5863 | 37800 | 0.2498 | 28831632 | | 0.2608 | 79.0021 | 38000 | 0.2486 | 28983144 | | 0.2627 | 79.4179 | 38200 | 0.2487 | 29136008 | | 0.244 | 79.8337 | 38400 | 0.2490 | 29288104 | | 0.269 | 80.2495 | 38600 | 0.2490 | 29440312 | | 0.2627 | 80.6653 | 38800 | 0.2491 | 29592888 | | 0.2656 | 81.0811 | 39000 | 0.2488 | 29745320 | | 0.2409 | 81.4969 | 39200 | 0.2488 | 29898600 | | 0.2619 | 81.9127 | 39400 | 0.2492 | 30050504 | | 0.2635 | 82.3285 | 39600 | 0.2492 | 30203576 | | 0.2449 | 82.7443 | 39800 | 0.2485 | 30356408 | | 0.2824 | 83.1601 | 40000 | 0.2493 | 30508240 | ### Framework versions - PEFT 0.15.1 - Transformers 4.51.3 - Pytorch 2.6.0+cu124 - Datasets 3.5.0 - Tokenizers 0.21.1