longformer-full_labels / README.md

Theoreticallyhugo

trainer: training complete at 2024-03-02 11:28:41.420619.

bf837ff verified almost 2 years ago

preview code

raw

history blame

22.7 kB

metadata

license: apache-2.0
base_model: allenai/longformer-base-4096
tags:
  - generated_from_trainer
datasets:
  - essays_su_g
metrics:
  - accuracy
model-index:
  - name: longformer-full_labels
    results:
      - task:
          name: Token Classification
          type: token-classification
        dataset:
          name: essays_su_g
          type: essays_su_g
          config: full_labels
          split: train[60%:80%]
          args: full_labels
        metrics:
          - name: Accuracy
            type: accuracy
            value: 0.8488955223880597

longformer-full_labels

This model is a fine-tuned version of allenai/longformer-base-4096 on the essays_su_g dataset. It achieves the following results on the evaluation set:

Loss: 0.6893
B-claim: {'precision': 0.6103151862464183, 'recall': 0.6283185840707964, 'f1-score': 0.6191860465116279, 'support': 339.0}
B-majorclaim: {'precision': 0.7816091954022989, 'recall': 0.85, 'f1-score': 0.8143712574850299, 'support': 160.0}
B-premise: {'precision': 0.761252446183953, 'recall': 0.8267800212539851, 'f1-score': 0.792664289353031, 'support': 941.0}
I-claim: {'precision': 0.636103781882146, 'recall': 0.6157939548744147, 'f1-score': 0.6257841228639411, 'support': 4698.0}
I-majorclaim: {'precision': 0.8445692883895131, 'recall': 0.8895463510848126, 'f1-score': 0.866474543707973, 'support': 2028.0}
I-premise: {'precision': 0.8711272247857613, 'recall': 0.8892402933853711, 'f1-score': 0.8800905730744897, 'support': 14861.0}
O: {'precision': 0.9305019305019305, 'recall': 0.897450587224291, 'f1-score': 0.9136774569845436, 'support': 10473.0}
Accuracy: 0.8489
Macro avg: {'precision': 0.7764970076274315, 'recall': 0.7995899702705245, 'f1-score': 0.7874640414258051, 'support': 33500.0}
Weighted avg: {'precision': 0.8489690555547813, 'recall': 0.8488955223880597, 'f1-score': 0.8486929780465652, 'support': 33500.0}

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

learning_rate: 2e-05
train_batch_size: 8
eval_batch_size: 8
seed: 42
optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
lr_scheduler_type: linear
num_epochs: 16

Training results

Training Loss	Epoch	Step	Validation Loss	B-claim	B-majorclaim	B-premise	I-claim	I-majorclaim	I-premise	O	Accuracy	Macro avg	Weighted avg
No log	1.0	41	0.7635	{'precision': 0.0, 'recall': 0.0, 'f1-score': 0.0, 'support': 339.0}	{'precision': 0.0, 'recall': 0.0, 'f1-score': 0.0, 'support': 160.0}	{'precision': 0.8299319727891157, 'recall': 0.12964930924548354, 'f1-score': 0.22426470588235295, 'support': 941.0}	{'precision': 0.423873640600725, 'recall': 0.348446147296722, 'f1-score': 0.38247663551401867, 'support': 4698.0}	{'precision': 0.5125673249551167, 'recall': 0.28155818540433925, 'f1-score': 0.3634627625716104, 'support': 2028.0}	{'precision': 0.8110747093209996, 'recall': 0.8496063521970257, 'f1-score': 0.8298935191271198, 'support': 14861.0}	{'precision': 0.7380952380952381, 'recall': 0.9027976701995608, 'f1-score': 0.8121805609242796, 'support': 10473.0}	0.7287	{'precision': 0.47364898368017067, 'recall': 0.3588653806204473, 'f1-score': 0.3731825977170545, 'support': 33500.0}	{'precision': 0.7043362259324342, 'recall': 0.7286865671641791, 'f1-score': 0.7040007584084623, 'support': 33500.0}
No log	2.0	82	0.5587	{'precision': 0.0, 'recall': 0.0, 'f1-score': 0.0, 'support': 339.0}	{'precision': 0.0, 'recall': 0.0, 'f1-score': 0.0, 'support': 160.0}	{'precision': 0.5819477434679335, 'recall': 0.7810839532412327, 'f1-score': 0.6669691470054446, 'support': 941.0}	{'precision': 0.559799275160301, 'recall': 0.4274159216687952, 'f1-score': 0.48473144236572113, 'support': 4698.0}	{'precision': 0.6312089971883786, 'recall': 0.6642011834319527, 'f1-score': 0.6472849591542528, 'support': 2028.0}	{'precision': 0.8200934579439252, 'recall': 0.9211358589596932, 'f1-score': 0.8676829461540899, 'support': 14861.0}	{'precision': 0.9105252442996743, 'recall': 0.8541010216747827, 'f1-score': 0.8814110459673844, 'support': 10473.0}	0.7977	{'precision': 0.5005106740086018, 'recall': 0.5211339912823509, 'f1-score': 0.506868505806699, 'support': 33500.0}	{'precision': 0.7815218337211824, 'recall': 0.7977313432835821, 'f1-score': 0.7863652038192075, 'support': 33500.0}
No log	3.0	123	0.5104	{'precision': 0.4175824175824176, 'recall': 0.22418879056047197, 'f1-score': 0.29174664107485604, 'support': 339.0}	{'precision': 0.8823529411764706, 'recall': 0.09375, 'f1-score': 0.1694915254237288, 'support': 160.0}	{'precision': 0.6546521374685667, 'recall': 0.8299681190223167, 'f1-score': 0.731958762886598, 'support': 941.0}	{'precision': 0.629277566539924, 'recall': 0.35227756492124307, 'f1-score': 0.45169213973799127, 'support': 4698.0}	{'precision': 0.7788510421962379, 'recall': 0.7554240631163708, 'f1-score': 0.7669586983729662, 'support': 2028.0}	{'precision': 0.7908845678318535, 'recall': 0.9469753044882578, 'f1-score': 0.8619200734956362, 'support': 14861.0}	{'precision': 0.9332098384274982, 'recall': 0.8658455074954645, 'f1-score': 0.8982664685487864, 'support': 10473.0}	0.8119	{'precision': 0.7266872158889954, 'recall': 0.5812041928005893, 'f1-score': 0.5960049013629376, 'support': 33500.0}	{'precision': 0.804819781883149, 'recall': 0.8119402985074626, 'f1-score': 0.7972768597429257, 'support': 33500.0}
No log	4.0	164	0.4638	{'precision': 0.6144200626959248, 'recall': 0.5781710914454278, 'f1-score': 0.5957446808510639, 'support': 339.0}	{'precision': 0.7910447761194029, 'recall': 0.6625, 'f1-score': 0.7210884353741497, 'support': 160.0}	{'precision': 0.7529411764705882, 'recall': 0.8161530286928799, 'f1-score': 0.7832738398776133, 'support': 941.0}	{'precision': 0.6121971950701233, 'recall': 0.6132396764580673, 'f1-score': 0.6127179923436835, 'support': 4698.0}	{'precision': 0.7631464580617123, 'recall': 0.8658777120315582, 'f1-score': 0.8112728112728113, 'support': 2028.0}	{'precision': 0.8770213344204376, 'recall': 0.8685821950070655, 'f1-score': 0.8727813651577132, 'support': 14861.0}	{'precision': 0.9097262667443214, 'recall': 0.8948725293612145, 'f1-score': 0.9022382671480144, 'support': 10473.0}	0.8354	{'precision': 0.7600710385117873, 'recall': 0.7570566047137447, 'f1-score': 0.7570167702892928, 'support': 33500.0}	{'precision': 0.8366601162482339, 'recall': 0.8354328358208956, 'f1-score': 0.8357536689114934, 'support': 33500.0}
No log	5.0	205	0.4939	{'precision': 0.6254295532646048, 'recall': 0.5368731563421829, 'f1-score': 0.5777777777777778, 'support': 339.0}	{'precision': 0.7636363636363637, 'recall': 0.7875, 'f1-score': 0.7753846153846152, 'support': 160.0}	{'precision': 0.7188898836168308, 'recall': 0.8533475026567482, 'f1-score': 0.7803692905733722, 'support': 941.0}	{'precision': 0.6303191489361702, 'recall': 0.5549169859514687, 'f1-score': 0.5902196060674665, 'support': 4698.0}	{'precision': 0.8303317535545024, 'recall': 0.863905325443787, 'f1-score': 0.8467858869018849, 'support': 2028.0}	{'precision': 0.8650767507740957, 'recall': 0.8835879146759976, 'f1-score': 0.8742343541944074, 'support': 14861.0}	{'precision': 0.904970481812988, 'recall': 0.9074763678029218, 'f1-score': 0.9062216924910609, 'support': 10473.0}	0.8390	{'precision': 0.7626648479422222, 'recall': 0.7696581789818724, 'f1-score': 0.7644276033415122, 'support': 33500.0}	{'precision': 0.8355072066989216, 'recall': 0.8389552238805971, 'f1-score': 0.8366342005564686, 'support': 33500.0}
No log	6.0	246	0.4979	{'precision': 0.6232876712328768, 'recall': 0.5368731563421829, 'f1-score': 0.5768621236133122, 'support': 339.0}	{'precision': 0.782051282051282, 'recall': 0.7625, 'f1-score': 0.7721518987341772, 'support': 160.0}	{'precision': 0.7490530303030303, 'recall': 0.8405951115834219, 'f1-score': 0.7921882824236354, 'support': 941.0}	{'precision': 0.6594274432379073, 'recall': 0.5687526607066837, 'f1-score': 0.6107428571428571, 'support': 4698.0}	{'precision': 0.8388861263053208, 'recall': 0.8318540433925049, 'f1-score': 0.8353552859618717, 'support': 2028.0}	{'precision': 0.8607000386050702, 'recall': 0.9001413094677343, 'f1-score': 0.8799789494457783, 'support': 14861.0}	{'precision': 0.9164661726494081, 'recall': 0.9092905566695312, 'f1-score': 0.912864263803681, 'support': 10473.0}	0.8464	{'precision': 0.7756959663406994, 'recall': 0.7642866911660084, 'f1-score': 0.7685919515893305, 'support': 33500.0}	{'precision': 0.8426730244638024, 'recall': 0.8463880597014926, 'f1-score': 0.843752680680015, 'support': 33500.0}
No log	7.0	287	0.5638	{'precision': 0.6418439716312057, 'recall': 0.5339233038348082, 'f1-score': 0.5829307568438002, 'support': 339.0}	{'precision': 0.7758620689655172, 'recall': 0.84375, 'f1-score': 0.8083832335329341, 'support': 160.0}	{'precision': 0.7203463203463204, 'recall': 0.8841657810839533, 'f1-score': 0.7938931297709925, 'support': 941.0}	{'precision': 0.7018554062699937, 'recall': 0.4670072371221797, 'f1-score': 0.560838445807771, 'support': 4698.0}	{'precision': 0.8422548555187115, 'recall': 0.8767258382642998, 'f1-score': 0.8591447209470888, 'support': 2028.0}	{'precision': 0.834623927006423, 'recall': 0.9356032568467801, 'f1-score': 0.882233502538071, 'support': 14861.0}	{'precision': 0.9340538376863805, 'recall': 0.8912441516279958, 'f1-score': 0.9121469754715138, 'support': 10473.0}	0.8465	{'precision': 0.7786914839177931, 'recall': 0.7760599383971453, 'f1-score': 0.7713672521303102, 'support': 33500.0}	{'precision': 0.8421095669207056, 'recall': 0.8465074626865672, 'f1-score': 0.8392525416975183, 'support': 33500.0}
No log	8.0	328	0.5493	{'precision': 0.5994550408719346, 'recall': 0.6489675516224189, 'f1-score': 0.623229461756374, 'support': 339.0}	{'precision': 0.7687861271676301, 'recall': 0.83125, 'f1-score': 0.7987987987987989, 'support': 160.0}	{'precision': 0.7347504621072088, 'recall': 0.844845908607864, 'f1-score': 0.7859614434008898, 'support': 941.0}	{'precision': 0.6259211096662332, 'recall': 0.6147296722009365, 'f1-score': 0.620274914089347, 'support': 4698.0}	{'precision': 0.8396803008932769, 'recall': 0.8806706114398422, 'f1-score': 0.8596871239470517, 'support': 2028.0}	{'precision': 0.8634082763115021, 'recall': 0.8915281609582127, 'f1-score': 0.8772429318678407, 'support': 14861.0}	{'precision': 0.9433210784313726, 'recall': 0.8819822400458321, 'f1-score': 0.9116210214655811, 'support': 10473.0}	0.8450	{'precision': 0.7679031993498798, 'recall': 0.7991391635535867, 'f1-score': 0.782402242189412, 'support': 33500.0}	{'precision': 0.8469129043625616, 'recall': 0.8450149253731343, 'f1-score': 0.8453815974816967, 'support': 33500.0}
No log	9.0	369	0.5795	{'precision': 0.6209150326797386, 'recall': 0.56047197640118, 'f1-score': 0.5891472868217054, 'support': 339.0}	{'precision': 0.810126582278481, 'recall': 0.8, 'f1-score': 0.8050314465408804, 'support': 160.0}	{'precision': 0.7478591817316841, 'recall': 0.8352816153028693, 'f1-score': 0.7891566265060241, 'support': 941.0}	{'precision': 0.6493925117778329, 'recall': 0.5574712643678161, 'f1-score': 0.5999312793494446, 'support': 4698.0}	{'precision': 0.8825079030558483, 'recall': 0.8259368836291914, 'f1-score': 0.8532857870606215, 'support': 2028.0}	{'precision': 0.8624886112195757, 'recall': 0.8917973218491353, 'f1-score': 0.8768981374268038, 'support': 14861.0}	{'precision': 0.8974550898203593, 'recall': 0.9158789267640599, 'f1-score': 0.9065734133547564, 'support': 10473.0}	0.8431	{'precision': 0.7815349875090741, 'recall': 0.7695482840448931, 'f1-score': 0.7742891395800339, 'support': 33500.0}	{'precision': 0.8388330863882018, 'recall': 0.8430746268656717, 'f1-score': 0.840184765735164, 'support': 33500.0}
No log	10.0	410	0.5960	{'precision': 0.6303630363036303, 'recall': 0.5634218289085545, 'f1-score': 0.5950155763239875, 'support': 339.0}	{'precision': 0.8, 'recall': 0.825, 'f1-score': 0.8123076923076924, 'support': 160.0}	{'precision': 0.7326642335766423, 'recall': 0.8533475026567482, 'f1-score': 0.7884143348060874, 'support': 941.0}	{'precision': 0.666921508664628, 'recall': 0.5570455512984248, 'f1-score': 0.6070517281373231, 'support': 4698.0}	{'precision': 0.8839779005524862, 'recall': 0.8678500986193294, 'f1-score': 0.8758397611346106, 'support': 2028.0}	{'precision': 0.851998491704374, 'recall': 0.9122535495592491, 'f1-score': 0.8810970656094628, 'support': 14861.0}	{'precision': 0.9333267385498071, 'recall': 0.900887997708393, 'f1-score': 0.9168205227869012, 'support': 10473.0}	0.8506	{'precision': 0.7856074156216526, 'recall': 0.7828295041072427, 'f1-score': 0.7823638115865806, 'support': 33500.0}	{'precision': 0.8475616436173209, 'recall': 0.8505970149253731, 'f1-score': 0.8476881875144826, 'support': 33500.0}
No log	11.0	451	0.5954	{'precision': 0.6235294117647059, 'recall': 0.6253687315634219, 'f1-score': 0.6244477172312225, 'support': 339.0}	{'precision': 0.7861271676300579, 'recall': 0.85, 'f1-score': 0.8168168168168167, 'support': 160.0}	{'precision': 0.7670286278381047, 'recall': 0.8257173219978746, 'f1-score': 0.7952917093142272, 'support': 941.0}	{'precision': 0.6513114392770554, 'recall': 0.6289910600255428, 'f1-score': 0.639956686518679, 'support': 4698.0}	{'precision': 0.8176156583629893, 'recall': 0.9063116370808678, 'f1-score': 0.8596819457436857, 'support': 2028.0}	{'precision': 0.8765022242878959, 'recall': 0.8882982302671422, 'f1-score': 0.8823608047590401, 'support': 14861.0}	{'precision': 0.9289099526066351, 'recall': 0.8983099398453165, 'f1-score': 0.9133537206931702, 'support': 10473.0}	0.8516	{'precision': 0.7787177831096349, 'recall': 0.8032852743971665, 'f1-score': 0.7902727715824059, 'support': 33500.0}	{'precision': 0.8516743266232883, 'recall': 0.8515522388059702, 'f1-score': 0.8513139373395039, 'support': 33500.0}
No log	12.0	492	0.6606	{'precision': 0.5913978494623656, 'recall': 0.6489675516224189, 'f1-score': 0.6188466947960619, 'support': 339.0}	{'precision': 0.7771428571428571, 'recall': 0.85, 'f1-score': 0.8119402985074626, 'support': 160.0}	{'precision': 0.7609126984126984, 'recall': 0.8150903294367694, 'f1-score': 0.7870702924576707, 'support': 941.0}	{'precision': 0.6260370134014039, 'recall': 0.6264367816091954, 'f1-score': 0.6262368337057134, 'support': 4698.0}	{'precision': 0.827490774907749, 'recall': 0.8846153846153846, 'f1-score': 0.8551000953288845, 'support': 2028.0}	{'precision': 0.8761073825503356, 'recall': 0.8784065675257385, 'f1-score': 0.8772554685662445, 'support': 14861.0}	{'precision': 0.9252161949685535, 'recall': 0.8989783252172252, 'f1-score': 0.9119085670008232, 'support': 10473.0}	0.8456	{'precision': 0.7691863958351375, 'recall': 0.8003564200038189, 'f1-score': 0.784051178623266, 'support': 33500.0}	{'precision': 0.8468579038739062, 'recall': 0.8456417910447761, 'f1-score': 0.8460850209295142, 'support': 33500.0}
0.3295	13.0	533	0.6498	{'precision': 0.6098901098901099, 'recall': 0.6548672566371682, 'f1-score': 0.631578947368421, 'support': 339.0}	{'precision': 0.7771428571428571, 'recall': 0.85, 'f1-score': 0.8119402985074626, 'support': 160.0}	{'precision': 0.7704590818363274, 'recall': 0.820403825717322, 'f1-score': 0.7946474523932063, 'support': 941.0}	{'precision': 0.6348204570184983, 'recall': 0.6209025117071094, 'f1-score': 0.6277843538146992, 'support': 4698.0}	{'precision': 0.8270746407046824, 'recall': 0.8796844181459567, 'f1-score': 0.8525686977299881, 'support': 2028.0}	{'precision': 0.8729010420123449, 'recall': 0.8850010093533409, 'f1-score': 0.8789093825180433, 'support': 14861.0}	{'precision': 0.9270216962524654, 'recall': 0.8975460708488494, 'f1-score': 0.912045796342114, 'support': 10473.0}	0.8473	{'precision': 0.7741871264081837, 'recall': 0.8012007274871067, 'f1-score': 0.7870678469534192, 'support': 33500.0}	{'precision': 0.8476618534036751, 'recall': 0.8472537313432836, 'f1-score': 0.8472670786725544, 'support': 33500.0}
0.3295	14.0	574	0.6825	{'precision': 0.6203966005665722, 'recall': 0.6460176991150443, 'f1-score': 0.6329479768786126, 'support': 339.0}	{'precision': 0.8023952095808383, 'recall': 0.8375, 'f1-score': 0.819571865443425, 'support': 160.0}	{'precision': 0.7741935483870968, 'recall': 0.8161530286928799, 'f1-score': 0.7946197620279358, 'support': 941.0}	{'precision': 0.6377252007814196, 'recall': 0.6253724989357173, 'f1-score': 0.631488447071467, 'support': 4698.0}	{'precision': 0.8424041646947468, 'recall': 0.8777120315581854, 'f1-score': 0.859695725670128, 'support': 2028.0}	{'precision': 0.8764517420905086, 'recall': 0.8835879146759976, 'f1-score': 0.880005361391281, 'support': 14861.0}	{'precision': 0.9212521874392378, 'recall': 0.904802826315287, 'f1-score': 0.9129534177946914, 'support': 10473.0}	0.8491	{'precision': 0.7821169505057742, 'recall': 0.7987351427561588, 'f1-score': 0.790183222325363, 'support': 33500.0}	{'precision': 0.8491004760503732, 'recall': 0.8491343283582089, 'f1-score': 0.8490374488030293, 'support': 33500.0}
0.3295	15.0	615	0.6853	{'precision': 0.6056338028169014, 'recall': 0.6342182890855457, 'f1-score': 0.6195965417867435, 'support': 339.0}	{'precision': 0.7894736842105263, 'recall': 0.84375, 'f1-score': 0.8157099697885197, 'support': 160.0}	{'precision': 0.7644749754661433, 'recall': 0.8278427205100957, 'f1-score': 0.7948979591836736, 'support': 941.0}	{'precision': 0.6346823324630113, 'recall': 0.6209025117071094, 'f1-score': 0.6277168065418549, 'support': 4698.0}	{'precision': 0.8516405135520685, 'recall': 0.8831360946745562, 'f1-score': 0.8671023965141612, 'support': 2028.0}	{'precision': 0.8722053181637783, 'recall': 0.8872888769261826, 'f1-score': 0.879682444377731, 'support': 14861.0}	{'precision': 0.9284868810416256, 'recall': 0.8987873579681085, 'f1-score': 0.9133957595458736, 'support': 10473.0}	0.8488	{'precision': 0.778085358244865, 'recall': 0.7994179786959427, 'f1-score': 0.7883002682483654, 'support': 33500.0}	{'precision': 0.8491267909945652, 'recall': 0.8488358208955223, 'f1-score': 0.8488061974719241, 'support': 33500.0}
0.3295	16.0	656	0.6893	{'precision': 0.6103151862464183, 'recall': 0.6283185840707964, 'f1-score': 0.6191860465116279, 'support': 339.0}	{'precision': 0.7816091954022989, 'recall': 0.85, 'f1-score': 0.8143712574850299, 'support': 160.0}	{'precision': 0.761252446183953, 'recall': 0.8267800212539851, 'f1-score': 0.792664289353031, 'support': 941.0}	{'precision': 0.636103781882146, 'recall': 0.6157939548744147, 'f1-score': 0.6257841228639411, 'support': 4698.0}	{'precision': 0.8445692883895131, 'recall': 0.8895463510848126, 'f1-score': 0.866474543707973, 'support': 2028.0}	{'precision': 0.8711272247857613, 'recall': 0.8892402933853711, 'f1-score': 0.8800905730744897, 'support': 14861.0}	{'precision': 0.9305019305019305, 'recall': 0.897450587224291, 'f1-score': 0.9136774569845436, 'support': 10473.0}	0.8489	{'precision': 0.7764970076274315, 'recall': 0.7995899702705245, 'f1-score': 0.7874640414258051, 'support': 33500.0}	{'precision': 0.8489690555547813, 'recall': 0.8488955223880597, 'f1-score': 0.8486929780465652, 'support': 33500.0}

Framework versions

Transformers 4.37.2
Pytorch 2.2.0+cu121
Datasets 2.17.0
Tokenizers 0.15.2