Hugging Face's logo

cutelemonlili
/
Qwen2.5-Math-7B-Instruct_lora_MATH_training_Qwen_QwQ_32B_Preview_checkpoint-500

PEFT
Safetensors
Model card Files
xet
Community
cutelemonlili commited on
Commit
cd512ae
·
verified ·
1 Parent(s): 2193665

Add files using upload-large-folder tool

Browse files
Files changed (19) hide show
  1. .gitattributes +1 -0
  2. README.md +202 -0
  3. adapter_config.json +34 -0
  4. adapter_model.safetensors +3 -0
  5. added_tokens.json +24 -0
  6. latest +1 -0
  7. merges.txt +0 -0
  8. rng_state_0.pth +3 -0
  9. rng_state_1.pth +3 -0
  10. rng_state_2.pth +3 -0
  11. rng_state_3.pth +3 -0
  12. scheduler.pt +3 -0
  13. special_tokens_map.json +31 -0
  14. tokenizer.json +3 -0
  15. tokenizer_config.json +208 -0
  16. trainer_state.json +3549 -0
  17. training_args.bin +3 -0
  18. vocab.json +0 -0
  19. zero_to_fp32.py +760 -0
.gitattributes CHANGED
@@ -33,3 +33,4 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
33
  *.zip filter=lfs diff=lfs merge=lfs -text
34
  *.zst filter=lfs diff=lfs merge=lfs -text
35
  *tfevents* filter=lfs diff=lfs merge=lfs -text
 
 
33
  *.zip filter=lfs diff=lfs merge=lfs -text
34
  *.zst filter=lfs diff=lfs merge=lfs -text
35
  *tfevents* filter=lfs diff=lfs merge=lfs -text
36
+ tokenizer.json filter=lfs diff=lfs merge=lfs -text
README.md ADDED
@@ -0,0 +1,202 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ base_model: Qwen/Qwen2.5-Math-7B-Instruct
3
+ library_name: peft
4
+ ---
5
+
6
+ # Model Card for Model ID
7
+
8
+ <!-- Provide a quick summary of what the model is/does. -->
9
+
10
+
11
+
12
+ ## Model Details
13
+
14
+ ### Model Description
15
+
16
+ <!-- Provide a longer summary of what this model is. -->
17
+
18
+
19
+
20
+ - **Developed by:** [More Information Needed]
21
+ - **Funded by [optional]:** [More Information Needed]
22
+ - **Shared by [optional]:** [More Information Needed]
23
+ - **Model type:** [More Information Needed]
24
+ - **Language(s) (NLP):** [More Information Needed]
25
+ - **License:** [More Information Needed]
26
+ - **Finetuned from model [optional]:** [More Information Needed]
27
+
28
+ ### Model Sources [optional]
29
+
30
+ <!-- Provide the basic links for the model. -->
31
+
32
+ - **Repository:** [More Information Needed]
33
+ - **Paper [optional]:** [More Information Needed]
34
+ - **Demo [optional]:** [More Information Needed]
35
+
36
+ ## Uses
37
+
38
+ <!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
39
+
40
+ ### Direct Use
41
+
42
+ <!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
43
+
44
+ [More Information Needed]
45
+
46
+ ### Downstream Use [optional]
47
+
48
+ <!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
49
+
50
+ [More Information Needed]
51
+
52
+ ### Out-of-Scope Use
53
+
54
+ <!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
55
+
56
+ [More Information Needed]
57
+
58
+ ## Bias, Risks, and Limitations
59
+
60
+ <!-- This section is meant to convey both technical and sociotechnical limitations. -->
61
+
62
+ [More Information Needed]
63
+
64
+ ### Recommendations
65
+
66
+ <!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
67
+
68
+ Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
69
+
70
+ ## How to Get Started with the Model
71
+
72
+ Use the code below to get started with the model.
73
+
74
+ [More Information Needed]
75
+
76
+ ## Training Details
77
+
78
+ ### Training Data
79
+
80
+ <!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
81
+
82
+ [More Information Needed]
83
+
84
+ ### Training Procedure
85
+
86
+ <!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
87
+
88
+ #### Preprocessing [optional]
89
+
90
+ [More Information Needed]
91
+
92
+
93
+ #### Training Hyperparameters
94
+
95
+ - **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
96
+
97
+ #### Speeds, Sizes, Times [optional]
98
+
99
+ <!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
100
+
101
+ [More Information Needed]
102
+
103
+ ## Evaluation
104
+
105
+ <!-- This section describes the evaluation protocols and provides the results. -->
106
+
107
+ ### Testing Data, Factors & Metrics
108
+
109
+ #### Testing Data
110
+
111
+ <!-- This should link to a Dataset Card if possible. -->
112
+
113
+ [More Information Needed]
114
+
115
+ #### Factors
116
+
117
+ <!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
118
+
119
+ [More Information Needed]
120
+
121
+ #### Metrics
122
+
123
+ <!-- These are the evaluation metrics being used, ideally with a description of why. -->
124
+
125
+ [More Information Needed]
126
+
127
+ ### Results
128
+
129
+ [More Information Needed]
130
+
131
+ #### Summary
132
+
133
+
134
+
135
+ ## Model Examination [optional]
136
+
137
+ <!-- Relevant interpretability work for the model goes here -->
138
+
139
+ [More Information Needed]
140
+
141
+ ## Environmental Impact
142
+
143
+ <!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
144
+
145
+ Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
146
+
147
+ - **Hardware Type:** [More Information Needed]
148
+ - **Hours used:** [More Information Needed]
149
+ - **Cloud Provider:** [More Information Needed]
150
+ - **Compute Region:** [More Information Needed]
151
+ - **Carbon Emitted:** [More Information Needed]
152
+
153
+ ## Technical Specifications [optional]
154
+
155
+ ### Model Architecture and Objective
156
+
157
+ [More Information Needed]
158
+
159
+ ### Compute Infrastructure
160
+
161
+ [More Information Needed]
162
+
163
+ #### Hardware
164
+
165
+ [More Information Needed]
166
+
167
+ #### Software
168
+
169
+ [More Information Needed]
170
+
171
+ ## Citation [optional]
172
+
173
+ <!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
174
+
175
+ **BibTeX:**
176
+
177
+ [More Information Needed]
178
+
179
+ **APA:**
180
+
181
+ [More Information Needed]
182
+
183
+ ## Glossary [optional]
184
+
185
+ <!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
186
+
187
+ [More Information Needed]
188
+
189
+ ## More Information [optional]
190
+
191
+ [More Information Needed]
192
+
193
+ ## Model Card Authors [optional]
194
+
195
+ [More Information Needed]
196
+
197
+ ## Model Card Contact
198
+
199
+ [More Information Needed]
200
+ ### Framework versions
201
+
202
+ - PEFT 0.12.0
adapter_config.json ADDED
@@ -0,0 +1,34 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "alpha_pattern": {},
3
+ "auto_mapping": null,
4
+ "base_model_name_or_path": "Qwen/Qwen2.5-Math-7B-Instruct",
5
+ "bias": "none",
6
+ "fan_in_fan_out": false,
7
+ "inference_mode": true,
8
+ "init_lora_weights": true,
9
+ "layer_replication": null,
10
+ "layers_pattern": null,
11
+ "layers_to_transform": null,
12
+ "loftq_config": {},
13
+ "lora_alpha": 16,
14
+ "lora_dropout": 0.0,
15
+ "megatron_config": null,
16
+ "megatron_core": "megatron.core",
17
+ "modules_to_save": null,
18
+ "peft_type": "LORA",
19
+ "r": 8,
20
+ "rank_pattern": {},
21
+ "revision": null,
22
+ "target_modules": [
23
+ "down_proj",
24
+ "q_proj",
25
+ "k_proj",
26
+ "o_proj",
27
+ "v_proj",
28
+ "gate_proj",
29
+ "up_proj"
30
+ ],
31
+ "task_type": "CAUSAL_LM",
32
+ "use_dora": false,
33
+ "use_rslora": false
34
+ }
adapter_model.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:2648d0a03ac3c7979ad26a18475e78e1c7942180ff7c76ef856f2b28ba8fd2b2
3
+ size 40422208
added_tokens.json ADDED
@@ -0,0 +1,24 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "</tool_call>": 151658,
3
+ "<tool_call>": 151657,
4
+ "<|box_end|>": 151649,
5
+ "<|box_start|>": 151648,
6
+ "<|endoftext|>": 151643,
7
+ "<|file_sep|>": 151664,
8
+ "<|fim_middle|>": 151660,
9
+ "<|fim_pad|>": 151662,
10
+ "<|fim_prefix|>": 151659,
11
+ "<|fim_suffix|>": 151661,
12
+ "<|im_end|>": 151645,
13
+ "<|im_start|>": 151644,
14
+ "<|image_pad|>": 151655,
15
+ "<|object_ref_end|>": 151647,
16
+ "<|object_ref_start|>": 151646,
17
+ "<|quad_end|>": 151651,
18
+ "<|quad_start|>": 151650,
19
+ "<|repo_name|>": 151663,
20
+ "<|video_pad|>": 151656,
21
+ "<|vision_end|>": 151653,
22
+ "<|vision_pad|>": 151654,
23
+ "<|vision_start|>": 151652
24
+ }
latest ADDED
@@ -0,0 +1 @@
 
 
1
+ global_step500
merges.txt ADDED
The diff for this file is too large to render. See raw diff
 
rng_state_0.pth ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:308f94f9a5c24e1bad5c393d56ae7af7782600f4e791d9c6ac35b22fff2105b6
3
+ size 15024
rng_state_1.pth ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:b056f3c23cb32dc77a2ec9e7651e0b64e4440e21f0fdf969b86bfc56a1cbdf06
3
+ size 15024
rng_state_2.pth ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:f3f8a05714bc528f4885a2816181652f2303b3e8150f89b56aaee6bec56aa520
3
+ size 15024
rng_state_3.pth ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:4f755bd3c330281961e5c03af9d10ce8c1e1678619d384f6f1fd5fd7dce2ff50
3
+ size 15024
scheduler.pt ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:ba21c9d2d086a96e35eaefbaf5f20c74cbba4792742079d64e7ddb91a4625f1b
3
+ size 1064
special_tokens_map.json ADDED
@@ -0,0 +1,31 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "additional_special_tokens": [
3
+ "<|im_start|>",
4
+ "<|im_end|>",
5
+ "<|object_ref_start|>",
6
+ "<|object_ref_end|>",
7
+ "<|box_start|>",
8
+ "<|box_end|>",
9
+ "<|quad_start|>",
10
+ "<|quad_end|>",
11
+ "<|vision_start|>",
12
+ "<|vision_end|>",
13
+ "<|vision_pad|>",
14
+ "<|image_pad|>",
15
+ "<|video_pad|>"
16
+ ],
17
+ "eos_token": {
18
+ "content": "<|im_end|>",
19
+ "lstrip": false,
20
+ "normalized": false,
21
+ "rstrip": false,
22
+ "single_word": false
23
+ },
24
+ "pad_token": {
25
+ "content": "<|endoftext|>",
26
+ "lstrip": false,
27
+ "normalized": false,
28
+ "rstrip": false,
29
+ "single_word": false
30
+ }
31
+ }
tokenizer.json ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:9c5ae00e602b8860cbd784ba82a8aa14e8feecec692e7076590d014d7b7fdafa
3
+ size 11421896
tokenizer_config.json ADDED
@@ -0,0 +1,208 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "add_bos_token": false,
3
+ "add_prefix_space": false,
4
+ "added_tokens_decoder": {
5
+ "151643": {
6
+ "content": "<|endoftext|>",
7
+ "lstrip": false,
8
+ "normalized": false,
9
+ "rstrip": false,
10
+ "single_word": false,
11
+ "special": true
12
+ },
13
+ "151644": {
14
+ "content": "<|im_start|>",
15
+ "lstrip": false,
16
+ "normalized": false,
17
+ "rstrip": false,
18
+ "single_word": false,
19
+ "special": true
20
+ },
21
+ "151645": {
22
+ "content": "<|im_end|>",
23
+ "lstrip": false,
24
+ "normalized": false,
25
+ "rstrip": false,
26
+ "single_word": false,
27
+ "special": true
28
+ },
29
+ "151646": {
30
+ "content": "<|object_ref_start|>",
31
+ "lstrip": false,
32
+ "normalized": false,
33
+ "rstrip": false,
34
+ "single_word": false,
35
+ "special": true
36
+ },
37
+ "151647": {
38
+ "content": "<|object_ref_end|>",
39
+ "lstrip": false,
40
+ "normalized": false,
41
+ "rstrip": false,
42
+ "single_word": false,
43
+ "special": true
44
+ },
45
+ "151648": {
46
+ "content": "<|box_start|>",
47
+ "lstrip": false,
48
+ "normalized": false,
49
+ "rstrip": false,
50
+ "single_word": false,
51
+ "special": true
52
+ },
53
+ "151649": {
54
+ "content": "<|box_end|>",
55
+ "lstrip": false,
56
+ "normalized": false,
57
+ "rstrip": false,
58
+ "single_word": false,
59
+ "special": true
60
+ },
61
+ "151650": {
62
+ "content": "<|quad_start|>",
63
+ "lstrip": false,
64
+ "normalized": false,
65
+ "rstrip": false,
66
+ "single_word": false,
67
+ "special": true
68
+ },
69
+ "151651": {
70
+ "content": "<|quad_end|>",
71
+ "lstrip": false,
72
+ "normalized": false,
73
+ "rstrip": false,
74
+ "single_word": false,
75
+ "special": true
76
+ },
77
+ "151652": {
78
+ "content": "<|vision_start|>",
79
+ "lstrip": false,
80
+ "normalized": false,
81
+ "rstrip": false,
82
+ "single_word": false,
83
+ "special": true
84
+ },
85
+ "151653": {
86
+ "content": "<|vision_end|>",
87
+ "lstrip": false,
88
+ "normalized": false,
89
+ "rstrip": false,
90
+ "single_word": false,
91
+ "special": true
92
+ },
93
+ "151654": {
94
+ "content": "<|vision_pad|>",
95
+ "lstrip": false,
96
+ "normalized": false,
97
+ "rstrip": false,
98
+ "single_word": false,
99
+ "special": true
100
+ },
101
+ "151655": {
102
+ "content": "<|image_pad|>",
103
+ "lstrip": false,
104
+ "normalized": false,
105
+ "rstrip": false,
106
+ "single_word": false,
107
+ "special": true
108
+ },
109
+ "151656": {
110
+ "content": "<|video_pad|>",
111
+ "lstrip": false,
112
+ "normalized": false,
113
+ "rstrip": false,
114
+ "single_word": false,
115
+ "special": true
116
+ },
117
+ "151657": {
118
+ "content": "<tool_call>",
119
+ "lstrip": false,
120
+ "normalized": false,
121
+ "rstrip": false,
122
+ "single_word": false,
123
+ "special": false
124
+ },
125
+ "151658": {
126
+ "content": "</tool_call>",
127
+ "lstrip": false,
128
+ "normalized": false,
129
+ "rstrip": false,
130
+ "single_word": false,
131
+ "special": false
132
+ },
133
+ "151659": {
134
+ "content": "<|fim_prefix|>",
135
+ "lstrip": false,
136
+ "normalized": false,
137
+ "rstrip": false,
138
+ "single_word": false,
139
+ "special": false
140
+ },
141
+ "151660": {
142
+ "content": "<|fim_middle|>",
143
+ "lstrip": false,
144
+ "normalized": false,
145
+ "rstrip": false,
146
+ "single_word": false,
147
+ "special": false
148
+ },
149
+ "151661": {
150
+ "content": "<|fim_suffix|>",
151
+ "lstrip": false,
152
+ "normalized": false,
153
+ "rstrip": false,
154
+ "single_word": false,
155
+ "special": false
156
+ },
157
+ "151662": {
158
+ "content": "<|fim_pad|>",
159
+ "lstrip": false,
160
+ "normalized": false,
161
+ "rstrip": false,
162
+ "single_word": false,
163
+ "special": false
164
+ },
165
+ "151663": {
166
+ "content": "<|repo_name|>",
167
+ "lstrip": false,
168
+ "normalized": false,
169
+ "rstrip": false,
170
+ "single_word": false,
171
+ "special": false
172
+ },
173
+ "151664": {
174
+ "content": "<|file_sep|>",
175
+ "lstrip": false,
176
+ "normalized": false,
177
+ "rstrip": false,
178
+ "single_word": false,
179
+ "special": false
180
+ }
181
+ },
182
+ "additional_special_tokens": [
183
+ "<|im_start|>",
184
+ "<|im_end|>",
185
+ "<|object_ref_start|>",
186
+ "<|object_ref_end|>",
187
+ "<|box_start|>",
188
+ "<|box_end|>",
189
+ "<|quad_start|>",
190
+ "<|quad_end|>",
191
+ "<|vision_start|>",
192
+ "<|vision_end|>",
193
+ "<|vision_pad|>",
194
+ "<|image_pad|>",
195
+ "<|video_pad|>"
196
+ ],
197
+ "bos_token": null,
198
+ "chat_template": "{%- if tools %}\n {{- '<|im_start|>system\\n' }}\n {%- if messages[0]['role'] == 'system' %}\n {{- messages[0]['content'] }}\n {%- else %}\n {{- 'Please reason step by step, and put your final answer within \\\\boxed{}.' }}\n {%- endif %}\n {{- \"\\n\\n# Tools\\n\\nYou may call one or more functions to assist with the user query.\\n\\nYou are provided with function signatures within <tools></tools> XML tags:\\n<tools>\" }}\n {%- for tool in tools %}\n {{- \"\\n\" }}\n {{- tool | tojson }}\n {%- endfor %}\n {{- \"\\n</tools>\\n\\nFor each function call, return a json object with function name and arguments within <tool_call></tool_call> XML tags:\\n<tool_call>\\n{\\\"name\\\": <function-name>, \\\"arguments\\\": <args-json-object>}\\n</tool_call><|im_end|>\\n\" }}\n{%- else %}\n {%- if messages[0]['role'] == 'system' %}\n {{- '<|im_start|>system\\n' + messages[0]['content'] + '<|im_end|>\\n' }}\n {%- else %}\n {{- '<|im_start|>system\\nPlease reason step by step, and put your final answer within \\\\boxed{}.<|im_end|>\\n' }}\n {%- endif %}\n{%- endif %}\n{%- for message in messages %}\n {%- if (message.role == \"user\") or (message.role == \"system\" and not loop.first) or (message.role == \"assistant\" and not message.tool_calls) %}\n {{- '<|im_start|>' + message.role + '\\n' + message.content + '<|im_end|>' + '\\n' }}\n {%- elif message.role == \"assistant\" %}\n {{- '<|im_start|>' + message.role }}\n {%- if message.content %}\n {{- '\\n' + message.content }}\n {%- endif %}\n {%- for tool_call in message.tool_calls %}\n {%- if tool_call.function is defined %}\n {%- set tool_call = tool_call.function %}\n {%- endif %}\n {{- '\\n<tool_call>\\n{\"name\": \"' }}\n {{- tool_call.name }}\n {{- '\", \"arguments\": ' }}\n {{- tool_call.arguments | tojson }}\n {{- '}\\n</tool_call>' }}\n {%- endfor %}\n {{- '<|im_end|>\\n' }}\n {%- elif message.role == \"tool\" %}\n {%- if (loop.index0 == 0) or (messages[loop.index0 - 1].role != \"tool\") %}\n {{- '<|im_start|>user' }}\n {%- endif %}\n {{- '\\n<tool_response>\\n' }}\n {{- message.content }}\n {{- '\\n</tool_response>' }}\n {%- if loop.last or (messages[loop.index0 + 1].role != \"tool\") %}\n {{- '<|im_end|>\\n' }}\n {%- endif %}\n {%- endif %}\n{%- endfor %}\n{%- if add_generation_prompt %}\n {{- '<|im_start|>assistant\\n' }}\n{%- endif %}\n",
199
+ "clean_up_tokenization_spaces": false,
200
+ "eos_token": "<|im_end|>",
201
+ "errors": "replace",
202
+ "model_max_length": 131072,
203
+ "pad_token": "<|endoftext|>",
204
+ "padding_side": "right",
205
+ "split_special_tokens": false,
206
+ "tokenizer_class": "Qwen2Tokenizer",
207
+ "unk_token": null
208
+ }
trainer_state.json ADDED
@@ -0,0 +1,3549 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "best_metric": null,
3
+ "best_model_checkpoint": null,
4
+ "epoch": 0.7496251874062968,
5
+ "eval_steps": 200,
6
+ "global_step": 500,
7
+ "is_hyper_param_search": false,
8
+ "is_local_process_zero": true,
9
+ "is_world_process_zero": true,
10
+ "log_history": [
11
+ {
12
+ "epoch": 0.0014992503748125937,
13
+ "grad_norm": 0.6480459224054971,
14
+ "learning_rate": 1.639344262295082e-06,
15
+ "loss": 1.2634,
16
+ "step": 1
17
+ },
18
+ {
19
+ "epoch": 0.0029985007496251873,
20
+ "grad_norm": 1.002261491564362,
21
+ "learning_rate": 3.278688524590164e-06,
22
+ "loss": 1.7059,
23
+ "step": 2
24
+ },
25
+ {
26
+ "epoch": 0.004497751124437781,
27
+ "grad_norm": 0.6235845014164785,
28
+ "learning_rate": 4.918032786885246e-06,
29
+ "loss": 1.2018,
30
+ "step": 3
31
+ },
32
+ {
33
+ "epoch": 0.005997001499250375,
34
+ "grad_norm": 0.7696945325835381,
35
+ "learning_rate": 6.557377049180328e-06,
36
+ "loss": 1.5389,
37
+ "step": 4
38
+ },
39
+ {
40
+ "epoch": 0.0074962518740629685,
41
+ "grad_norm": 0.7725389217188828,
42
+ "learning_rate": 8.196721311475409e-06,
43
+ "loss": 1.3489,
44
+ "step": 5
45
+ },
46
+ {
47
+ "epoch": 0.008995502248875561,
48
+ "grad_norm": 1.023311996781376,
49
+ "learning_rate": 9.836065573770493e-06,
50
+ "loss": 1.9915,
51
+ "step": 6
52
+ },
53
+ {
54
+ "epoch": 0.010494752623688156,
55
+ "grad_norm": 0.7234320068186162,
56
+ "learning_rate": 1.1475409836065575e-05,
57
+ "loss": 1.4321,
58
+ "step": 7
59
+ },
60
+ {
61
+ "epoch": 0.01199400299850075,
62
+ "grad_norm": 0.8106320449229776,
63
+ "learning_rate": 1.3114754098360657e-05,
64
+ "loss": 1.6013,
65
+ "step": 8
66
+ },
67
+ {
68
+ "epoch": 0.013493253373313344,
69
+ "grad_norm": 0.6614103055476157,
70
+ "learning_rate": 1.4754098360655739e-05,
71
+ "loss": 1.2124,
72
+ "step": 9
73
+ },
74
+ {
75
+ "epoch": 0.014992503748125937,
76
+ "grad_norm": 1.177719371372819,
77
+ "learning_rate": 1.6393442622950818e-05,
78
+ "loss": 1.8517,
79
+ "step": 10
80
+ },
81
+ {
82
+ "epoch": 0.01649175412293853,
83
+ "grad_norm": 0.8848406229059239,
84
+ "learning_rate": 1.8032786885245903e-05,
85
+ "loss": 1.5112,
86
+ "step": 11
87
+ },
88
+ {
89
+ "epoch": 0.017991004497751123,
90
+ "grad_norm": 0.9014496173514771,
91
+ "learning_rate": 1.9672131147540985e-05,
92
+ "loss": 1.535,
93
+ "step": 12
94
+ },
95
+ {
96
+ "epoch": 0.019490254872563718,
97
+ "grad_norm": 0.7836410412317578,
98
+ "learning_rate": 2.1311475409836064e-05,
99
+ "loss": 1.1882,
100
+ "step": 13
101
+ },
102
+ {
103
+ "epoch": 0.020989505247376312,
104
+ "grad_norm": 1.0481927081240034,
105
+ "learning_rate": 2.295081967213115e-05,
106
+ "loss": 1.519,
107
+ "step": 14
108
+ },
109
+ {
110
+ "epoch": 0.022488755622188907,
111
+ "grad_norm": 0.8252752830517661,
112
+ "learning_rate": 2.459016393442623e-05,
113
+ "loss": 1.2829,
114
+ "step": 15
115
+ },
116
+ {
117
+ "epoch": 0.0239880059970015,
118
+ "grad_norm": 1.4337440260538479,
119
+ "learning_rate": 2.6229508196721314e-05,
120
+ "loss": 1.8454,
121
+ "step": 16
122
+ },
123
+ {
124
+ "epoch": 0.025487256371814093,
125
+ "grad_norm": 1.1085278892757038,
126
+ "learning_rate": 2.7868852459016392e-05,
127
+ "loss": 1.4827,
128
+ "step": 17
129
+ },
130
+ {
131
+ "epoch": 0.026986506746626688,
132
+ "grad_norm": 0.9726032316099864,
133
+ "learning_rate": 2.9508196721311478e-05,
134
+ "loss": 1.3396,
135
+ "step": 18
136
+ },
137
+ {
138
+ "epoch": 0.02848575712143928,
139
+ "grad_norm": 1.134745439965142,
140
+ "learning_rate": 3.114754098360656e-05,
141
+ "loss": 1.3275,
142
+ "step": 19
143
+ },
144
+ {
145
+ "epoch": 0.029985007496251874,
146
+ "grad_norm": 1.3783626004268938,
147
+ "learning_rate": 3.2786885245901635e-05,
148
+ "loss": 1.4666,
149
+ "step": 20
150
+ },
151
+ {
152
+ "epoch": 0.031484257871064465,
153
+ "grad_norm": 1.42842588053584,
154
+ "learning_rate": 3.442622950819672e-05,
155
+ "loss": 1.4528,
156
+ "step": 21
157
+ },
158
+ {
159
+ "epoch": 0.03298350824587706,
160
+ "grad_norm": 1.5295744375884153,
161
+ "learning_rate": 3.6065573770491806e-05,
162
+ "loss": 1.3789,
163
+ "step": 22
164
+ },
165
+ {
166
+ "epoch": 0.034482758620689655,
167
+ "grad_norm": 1.3262347118296849,
168
+ "learning_rate": 3.7704918032786885e-05,
169
+ "loss": 1.29,
170
+ "step": 23
171
+ },
172
+ {
173
+ "epoch": 0.035982008995502246,
174
+ "grad_norm": 0.9094034688552372,
175
+ "learning_rate": 3.934426229508197e-05,
176
+ "loss": 1.0371,
177
+ "step": 24
178
+ },
179
+ {
180
+ "epoch": 0.037481259370314844,
181
+ "grad_norm": 1.1737967227100512,
182
+ "learning_rate": 4.098360655737705e-05,
183
+ "loss": 1.1038,
184
+ "step": 25
185
+ },
186
+ {
187
+ "epoch": 0.038980509745127435,
188
+ "grad_norm": 1.056641879506561,
189
+ "learning_rate": 4.262295081967213e-05,
190
+ "loss": 0.9753,
191
+ "step": 26
192
+ },
193
+ {
194
+ "epoch": 0.04047976011994003,
195
+ "grad_norm": 0.7823073229632485,
196
+ "learning_rate": 4.426229508196721e-05,
197
+ "loss": 1.0282,
198
+ "step": 27
199
+ },
200
+ {
201
+ "epoch": 0.041979010494752625,
202
+ "grad_norm": 0.44538213854923814,
203
+ "learning_rate": 4.59016393442623e-05,
204
+ "loss": 0.7663,
205
+ "step": 28
206
+ },
207
+ {
208
+ "epoch": 0.043478260869565216,
209
+ "grad_norm": 0.4381462092753938,
210
+ "learning_rate": 4.754098360655738e-05,
211
+ "loss": 0.7985,
212
+ "step": 29
213
+ },
214
+ {
215
+ "epoch": 0.044977511244377814,
216
+ "grad_norm": 0.4039939547366716,
217
+ "learning_rate": 4.918032786885246e-05,
218
+ "loss": 0.7191,
219
+ "step": 30
220
+ },
221
+ {
222
+ "epoch": 0.046476761619190406,
223
+ "grad_norm": 0.31255508798384235,
224
+ "learning_rate": 5.081967213114754e-05,
225
+ "loss": 0.7129,
226
+ "step": 31
227
+ },
228
+ {
229
+ "epoch": 0.047976011994003,
230
+ "grad_norm": 0.36801599741523594,
231
+ "learning_rate": 5.245901639344263e-05,
232
+ "loss": 0.8943,
233
+ "step": 32
234
+ },
235
+ {
236
+ "epoch": 0.049475262368815595,
237
+ "grad_norm": 0.23995271490026998,
238
+ "learning_rate": 5.409836065573771e-05,
239
+ "loss": 0.6516,
240
+ "step": 33
241
+ },
242
+ {
243
+ "epoch": 0.050974512743628186,
244
+ "grad_norm": 0.31423520475526784,
245
+ "learning_rate": 5.5737704918032785e-05,
246
+ "loss": 0.6279,
247
+ "step": 34
248
+ },
249
+ {
250
+ "epoch": 0.05247376311844078,
251
+ "grad_norm": 0.2790356075550021,
252
+ "learning_rate": 5.737704918032787e-05,
253
+ "loss": 0.7248,
254
+ "step": 35
255
+ },
256
+ {
257
+ "epoch": 0.053973013493253376,
258
+ "grad_norm": 0.17988842758193976,
259
+ "learning_rate": 5.9016393442622956e-05,
260
+ "loss": 0.6779,
261
+ "step": 36
262
+ },
263
+ {
264
+ "epoch": 0.05547226386806597,
265
+ "grad_norm": 0.1935776869878135,
266
+ "learning_rate": 6.0655737704918034e-05,
267
+ "loss": 0.7249,
268
+ "step": 37
269
+ },
270
+ {
271
+ "epoch": 0.05697151424287856,
272
+ "grad_norm": 0.20577959928691567,
273
+ "learning_rate": 6.229508196721313e-05,
274
+ "loss": 0.7901,
275
+ "step": 38
276
+ },
277
+ {
278
+ "epoch": 0.05847076461769116,
279
+ "grad_norm": 0.1920317677722405,
280
+ "learning_rate": 6.39344262295082e-05,
281
+ "loss": 0.7587,
282
+ "step": 39
283
+ },
284
+ {
285
+ "epoch": 0.05997001499250375,
286
+ "grad_norm": 0.19588448674722603,
287
+ "learning_rate": 6.557377049180327e-05,
288
+ "loss": 0.6527,
289
+ "step": 40
290
+ },
291
+ {
292
+ "epoch": 0.06146926536731634,
293
+ "grad_norm": 0.19117240884440398,
294
+ "learning_rate": 6.721311475409836e-05,
295
+ "loss": 0.7422,
296
+ "step": 41
297
+ },
298
+ {
299
+ "epoch": 0.06296851574212893,
300
+ "grad_norm": 0.2126112343873058,
301
+ "learning_rate": 6.885245901639344e-05,
302
+ "loss": 0.6567,
303
+ "step": 42
304
+ },
305
+ {
306
+ "epoch": 0.06446776611694154,
307
+ "grad_norm": 0.1867157531444841,
308
+ "learning_rate": 7.049180327868853e-05,
309
+ "loss": 0.6655,
310
+ "step": 43
311
+ },
312
+ {
313
+ "epoch": 0.06596701649175413,
314
+ "grad_norm": 0.1766702070625266,
315
+ "learning_rate": 7.213114754098361e-05,
316
+ "loss": 0.6048,
317
+ "step": 44
318
+ },
319
+ {
320
+ "epoch": 0.06746626686656672,
321
+ "grad_norm": 0.21635503382268856,
322
+ "learning_rate": 7.377049180327869e-05,
323
+ "loss": 0.7181,
324
+ "step": 45
325
+ },
326
+ {
327
+ "epoch": 0.06896551724137931,
328
+ "grad_norm": 0.22720549019270686,
329
+ "learning_rate": 7.540983606557377e-05,
330
+ "loss": 0.7293,
331
+ "step": 46
332
+ },
333
+ {
334
+ "epoch": 0.0704647676161919,
335
+ "grad_norm": 0.20178392044030222,
336
+ "learning_rate": 7.704918032786885e-05,
337
+ "loss": 0.6708,
338
+ "step": 47
339
+ },
340
+ {
341
+ "epoch": 0.07196401799100449,
342
+ "grad_norm": 0.1866069924894252,
343
+ "learning_rate": 7.868852459016394e-05,
344
+ "loss": 0.569,
345
+ "step": 48
346
+ },
347
+ {
348
+ "epoch": 0.0734632683658171,
349
+ "grad_norm": 0.20423207217060818,
350
+ "learning_rate": 8.032786885245902e-05,
351
+ "loss": 0.6823,
352
+ "step": 49
353
+ },
354
+ {
355
+ "epoch": 0.07496251874062969,
356
+ "grad_norm": 0.27146563006319185,
357
+ "learning_rate": 8.19672131147541e-05,
358
+ "loss": 0.765,
359
+ "step": 50
360
+ },
361
+ {
362
+ "epoch": 0.07646176911544228,
363
+ "grad_norm": 0.18688552348259718,
364
+ "learning_rate": 8.360655737704919e-05,
365
+ "loss": 0.6836,
366
+ "step": 51
367
+ },
368
+ {
369
+ "epoch": 0.07796101949025487,
370
+ "grad_norm": 0.208689035179473,
371
+ "learning_rate": 8.524590163934426e-05,
372
+ "loss": 0.6735,
373
+ "step": 52
374
+ },
375
+ {
376
+ "epoch": 0.07946026986506746,
377
+ "grad_norm": 0.17717804777290422,
378
+ "learning_rate": 8.688524590163935e-05,
379
+ "loss": 0.657,
380
+ "step": 53
381
+ },
382
+ {
383
+ "epoch": 0.08095952023988005,
384
+ "grad_norm": 0.14649881728810155,
385
+ "learning_rate": 8.852459016393443e-05,
386
+ "loss": 0.5488,
387
+ "step": 54
388
+ },
389
+ {
390
+ "epoch": 0.08245877061469266,
391
+ "grad_norm": 0.190417488803657,
392
+ "learning_rate": 9.016393442622952e-05,
393
+ "loss": 0.6301,
394
+ "step": 55
395
+ },
396
+ {
397
+ "epoch": 0.08395802098950525,
398
+ "grad_norm": 0.1698375240514099,
399
+ "learning_rate": 9.18032786885246e-05,
400
+ "loss": 0.6596,
401
+ "step": 56
402
+ },
403
+ {
404
+ "epoch": 0.08545727136431784,
405
+ "grad_norm": 0.1962769006488063,
406
+ "learning_rate": 9.344262295081968e-05,
407
+ "loss": 0.6707,
408
+ "step": 57
409
+ },
410
+ {
411
+ "epoch": 0.08695652173913043,
412
+ "grad_norm": 0.175718028167132,
413
+ "learning_rate": 9.508196721311476e-05,
414
+ "loss": 0.6894,
415
+ "step": 58
416
+ },
417
+ {
418
+ "epoch": 0.08845577211394302,
419
+ "grad_norm": 0.14251257889523739,
420
+ "learning_rate": 9.672131147540983e-05,
421
+ "loss": 0.4608,
422
+ "step": 59
423
+ },
424
+ {
425
+ "epoch": 0.08995502248875563,
426
+ "grad_norm": 0.1546737314752146,
427
+ "learning_rate": 9.836065573770493e-05,
428
+ "loss": 0.5651,
429
+ "step": 60
430
+ },
431
+ {
432
+ "epoch": 0.09145427286356822,
433
+ "grad_norm": 0.16119270097083238,
434
+ "learning_rate": 0.0001,
435
+ "loss": 0.5859,
436
+ "step": 61
437
+ },
438
+ {
439
+ "epoch": 0.09295352323838081,
440
+ "grad_norm": 0.17202976487555255,
441
+ "learning_rate": 9.999993444041447e-05,
442
+ "loss": 0.6663,
443
+ "step": 62
444
+ },
445
+ {
446
+ "epoch": 0.0944527736131934,
447
+ "grad_norm": 0.1744041549987411,
448
+ "learning_rate": 9.999973776182981e-05,
449
+ "loss": 0.5656,
450
+ "step": 63
451
+ },
452
+ {
453
+ "epoch": 0.095952023988006,
454
+ "grad_norm": 0.19578738914968555,
455
+ "learning_rate": 9.999940996476175e-05,
456
+ "loss": 0.6668,
457
+ "step": 64
458
+ },
459
+ {
460
+ "epoch": 0.09745127436281859,
461
+ "grad_norm": 0.13866011716797835,
462
+ "learning_rate": 9.999895105006994e-05,
463
+ "loss": 0.4206,
464
+ "step": 65
465
+ },
466
+ {
467
+ "epoch": 0.09895052473763119,
468
+ "grad_norm": 0.17606665683631137,
469
+ "learning_rate": 9.999836101895783e-05,
470
+ "loss": 0.608,
471
+ "step": 66
472
+ },
473
+ {
474
+ "epoch": 0.10044977511244378,
475
+ "grad_norm": 0.11399753463035138,
476
+ "learning_rate": 9.999763987297265e-05,
477
+ "loss": 0.4107,
478
+ "step": 67
479
+ },
480
+ {
481
+ "epoch": 0.10194902548725637,
482
+ "grad_norm": 0.1643566923671948,
483
+ "learning_rate": 9.999678761400561e-05,
484
+ "loss": 0.545,
485
+ "step": 68
486
+ },
487
+ {
488
+ "epoch": 0.10344827586206896,
489
+ "grad_norm": 0.15684085393275743,
490
+ "learning_rate": 9.99958042442916e-05,
491
+ "loss": 0.5278,
492
+ "step": 69
493
+ },
494
+ {
495
+ "epoch": 0.10494752623688156,
496
+ "grad_norm": 0.1779073694133138,
497
+ "learning_rate": 9.99946897664094e-05,
498
+ "loss": 0.5174,
499
+ "step": 70
500
+ },
501
+ {
502
+ "epoch": 0.10644677661169415,
503
+ "grad_norm": 0.12935127762781468,
504
+ "learning_rate": 9.999344418328162e-05,
505
+ "loss": 0.43,
506
+ "step": 71
507
+ },
508
+ {
509
+ "epoch": 0.10794602698650675,
510
+ "grad_norm": 0.2110979324649629,
511
+ "learning_rate": 9.999206749817463e-05,
512
+ "loss": 0.6861,
513
+ "step": 72
514
+ },
515
+ {
516
+ "epoch": 0.10944527736131934,
517
+ "grad_norm": 0.1936555889752225,
518
+ "learning_rate": 9.999055971469864e-05,
519
+ "loss": 0.5476,
520
+ "step": 73
521
+ },
522
+ {
523
+ "epoch": 0.11094452773613193,
524
+ "grad_norm": 0.20701659334369552,
525
+ "learning_rate": 9.998892083680764e-05,
526
+ "loss": 0.6848,
527
+ "step": 74
528
+ },
529
+ {
530
+ "epoch": 0.11244377811094453,
531
+ "grad_norm": 0.16610817248915638,
532
+ "learning_rate": 9.998715086879938e-05,
533
+ "loss": 0.5305,
534
+ "step": 75
535
+ },
536
+ {
537
+ "epoch": 0.11394302848575712,
538
+ "grad_norm": 0.17520351205683088,
539
+ "learning_rate": 9.99852498153154e-05,
540
+ "loss": 0.517,
541
+ "step": 76
542
+ },
543
+ {
544
+ "epoch": 0.11544227886056972,
545
+ "grad_norm": 0.1333804777211274,
546
+ "learning_rate": 9.998321768134102e-05,
547
+ "loss": 0.4761,
548
+ "step": 77
549
+ },
550
+ {
551
+ "epoch": 0.11694152923538231,
552
+ "grad_norm": 0.17827128088875116,
553
+ "learning_rate": 9.998105447220523e-05,
554
+ "loss": 0.4646,
555
+ "step": 78
556
+ },
557
+ {
558
+ "epoch": 0.1184407796101949,
559
+ "grad_norm": 0.1898191486639819,
560
+ "learning_rate": 9.997876019358084e-05,
561
+ "loss": 0.6011,
562
+ "step": 79
563
+ },
564
+ {
565
+ "epoch": 0.1199400299850075,
566
+ "grad_norm": 0.16388344609996736,
567
+ "learning_rate": 9.997633485148427e-05,
568
+ "loss": 0.4767,
569
+ "step": 80
570
+ },
571
+ {
572
+ "epoch": 0.12143928035982009,
573
+ "grad_norm": 0.15961092951225977,
574
+ "learning_rate": 9.997377845227576e-05,
575
+ "loss": 0.4282,
576
+ "step": 81
577
+ },
578
+ {
579
+ "epoch": 0.12293853073463268,
580
+ "grad_norm": 0.1751526879417404,
581
+ "learning_rate": 9.997109100265911e-05,
582
+ "loss": 0.5367,
583
+ "step": 82
584
+ },
585
+ {
586
+ "epoch": 0.12443778110944528,
587
+ "grad_norm": 0.1578787924131897,
588
+ "learning_rate": 9.99682725096819e-05,
589
+ "loss": 0.4191,
590
+ "step": 83
591
+ },
592
+ {
593
+ "epoch": 0.12593703148425786,
594
+ "grad_norm": 0.14912570536521574,
595
+ "learning_rate": 9.996532298073525e-05,
596
+ "loss": 0.4145,
597
+ "step": 84
598
+ },
599
+ {
600
+ "epoch": 0.12743628185907047,
601
+ "grad_norm": 0.16551987182072345,
602
+ "learning_rate": 9.996224242355399e-05,
603
+ "loss": 0.3979,
604
+ "step": 85
605
+ },
606
+ {
607
+ "epoch": 0.12893553223388307,
608
+ "grad_norm": 0.20627144163236044,
609
+ "learning_rate": 9.99590308462165e-05,
610
+ "loss": 0.4774,
611
+ "step": 86
612
+ },
613
+ {
614
+ "epoch": 0.13043478260869565,
615
+ "grad_norm": 0.17621912768125048,
616
+ "learning_rate": 9.995568825714479e-05,
617
+ "loss": 0.4261,
618
+ "step": 87
619
+ },
620
+ {
621
+ "epoch": 0.13193403298350825,
622
+ "grad_norm": 0.17245900092543753,
623
+ "learning_rate": 9.995221466510439e-05,
624
+ "loss": 0.393,
625
+ "step": 88
626
+ },
627
+ {
628
+ "epoch": 0.13343328335832083,
629
+ "grad_norm": 0.1575650027375934,
630
+ "learning_rate": 9.99486100792044e-05,
631
+ "loss": 0.4537,
632
+ "step": 89
633
+ },
634
+ {
635
+ "epoch": 0.13493253373313344,
636
+ "grad_norm": 0.1871804347088243,
637
+ "learning_rate": 9.994487450889742e-05,
638
+ "loss": 0.517,
639
+ "step": 90
640
+ },
641
+ {
642
+ "epoch": 0.136431784107946,
643
+ "grad_norm": 0.20988683934292762,
644
+ "learning_rate": 9.994100796397954e-05,
645
+ "loss": 0.502,
646
+ "step": 91
647
+ },
648
+ {
649
+ "epoch": 0.13793103448275862,
650
+ "grad_norm": 0.17193303094730353,
651
+ "learning_rate": 9.993701045459033e-05,
652
+ "loss": 0.4646,
653
+ "step": 92
654
+ },
655
+ {
656
+ "epoch": 0.13943028485757122,
657
+ "grad_norm": 0.1802261880462562,
658
+ "learning_rate": 9.993288199121283e-05,
659
+ "loss": 0.3887,
660
+ "step": 93
661
+ },
662
+ {
663
+ "epoch": 0.1409295352323838,
664
+ "grad_norm": 0.1858715725970327,
665
+ "learning_rate": 9.992862258467339e-05,
666
+ "loss": 0.4223,
667
+ "step": 94
668
+ },
669
+ {
670
+ "epoch": 0.1424287856071964,
671
+ "grad_norm": 0.2089799533634207,
672
+ "learning_rate": 9.992423224614185e-05,
673
+ "loss": 0.4704,
674
+ "step": 95
675
+ },
676
+ {
677
+ "epoch": 0.14392803598200898,
678
+ "grad_norm": 0.19387439159050426,
679
+ "learning_rate": 9.991971098713136e-05,
680
+ "loss": 0.5326,
681
+ "step": 96
682
+ },
683
+ {
684
+ "epoch": 0.1454272863568216,
685
+ "grad_norm": 0.17540287484028097,
686
+ "learning_rate": 9.991505881949837e-05,
687
+ "loss": 0.486,
688
+ "step": 97
689
+ },
690
+ {
691
+ "epoch": 0.1469265367316342,
692
+ "grad_norm": 0.18352832628580004,
693
+ "learning_rate": 9.991027575544265e-05,
694
+ "loss": 0.4695,
695
+ "step": 98
696
+ },
697
+ {
698
+ "epoch": 0.14842578710644677,
699
+ "grad_norm": 0.17778807224046753,
700
+ "learning_rate": 9.990536180750725e-05,
701
+ "loss": 0.3978,
702
+ "step": 99
703
+ },
704
+ {
705
+ "epoch": 0.14992503748125938,
706
+ "grad_norm": 0.18825789188216346,
707
+ "learning_rate": 9.990031698857841e-05,
708
+ "loss": 0.5293,
709
+ "step": 100
710
+ },
711
+ {
712
+ "epoch": 0.15142428785607195,
713
+ "grad_norm": 0.17034812176222888,
714
+ "learning_rate": 9.989514131188559e-05,
715
+ "loss": 0.4371,
716
+ "step": 101
717
+ },
718
+ {
719
+ "epoch": 0.15292353823088456,
720
+ "grad_norm": 0.17425360317449376,
721
+ "learning_rate": 9.988983479100139e-05,
722
+ "loss": 0.4473,
723
+ "step": 102
724
+ },
725
+ {
726
+ "epoch": 0.15442278860569716,
727
+ "grad_norm": 0.14790150868922555,
728
+ "learning_rate": 9.988439743984154e-05,
729
+ "loss": 0.3456,
730
+ "step": 103
731
+ },
732
+ {
733
+ "epoch": 0.15592203898050974,
734
+ "grad_norm": 0.1665767949204408,
735
+ "learning_rate": 9.987882927266487e-05,
736
+ "loss": 0.4144,
737
+ "step": 104
738
+ },
739
+ {
740
+ "epoch": 0.15742128935532235,
741
+ "grad_norm": 0.16332701910274577,
742
+ "learning_rate": 9.987313030407323e-05,
743
+ "loss": 0.4261,
744
+ "step": 105
745
+ },
746
+ {
747
+ "epoch": 0.15892053973013492,
748
+ "grad_norm": 0.17586519504691273,
749
+ "learning_rate": 9.986730054901153e-05,
750
+ "loss": 0.4841,
751
+ "step": 106
752
+ },
753
+ {
754
+ "epoch": 0.16041979010494753,
755
+ "grad_norm": 0.16205425976408627,
756
+ "learning_rate": 9.98613400227676e-05,
757
+ "loss": 0.4223,
758
+ "step": 107
759
+ },
760
+ {
761
+ "epoch": 0.1619190404797601,
762
+ "grad_norm": 0.15623786202509718,
763
+ "learning_rate": 9.985524874097224e-05,
764
+ "loss": 0.3693,
765
+ "step": 108
766
+ },
767
+ {
768
+ "epoch": 0.1634182908545727,
769
+ "grad_norm": 0.164145279114984,
770
+ "learning_rate": 9.984902671959911e-05,
771
+ "loss": 0.3768,
772
+ "step": 109
773
+ },
774
+ {
775
+ "epoch": 0.16491754122938532,
776
+ "grad_norm": 0.22244399379247035,
777
+ "learning_rate": 9.984267397496474e-05,
778
+ "loss": 0.4043,
779
+ "step": 110
780
+ },
781
+ {
782
+ "epoch": 0.1664167916041979,
783
+ "grad_norm": 0.19422715657443626,
784
+ "learning_rate": 9.983619052372848e-05,
785
+ "loss": 0.3957,
786
+ "step": 111
787
+ },
788
+ {
789
+ "epoch": 0.1679160419790105,
790
+ "grad_norm": 0.18763312575718583,
791
+ "learning_rate": 9.982957638289239e-05,
792
+ "loss": 0.4464,
793
+ "step": 112
794
+ },
795
+ {
796
+ "epoch": 0.16941529235382308,
797
+ "grad_norm": 0.21251049757841928,
798
+ "learning_rate": 9.982283156980132e-05,
799
+ "loss": 0.41,
800
+ "step": 113
801
+ },
802
+ {
803
+ "epoch": 0.17091454272863568,
804
+ "grad_norm": 0.1796936170293695,
805
+ "learning_rate": 9.981595610214275e-05,
806
+ "loss": 0.4376,
807
+ "step": 114
808
+ },
809
+ {
810
+ "epoch": 0.1724137931034483,
811
+ "grad_norm": 0.175685383349515,
812
+ "learning_rate": 9.980894999794678e-05,
813
+ "loss": 0.4391,
814
+ "step": 115
815
+ },
816
+ {
817
+ "epoch": 0.17391304347826086,
818
+ "grad_norm": 0.17656439408156688,
819
+ "learning_rate": 9.980181327558609e-05,
820
+ "loss": 0.4314,
821
+ "step": 116
822
+ },
823
+ {
824
+ "epoch": 0.17541229385307347,
825
+ "grad_norm": 0.2117047330447647,
826
+ "learning_rate": 9.979454595377594e-05,
827
+ "loss": 0.4373,
828
+ "step": 117
829
+ },
830
+ {
831
+ "epoch": 0.17691154422788605,
832
+ "grad_norm": 0.18946879886598905,
833
+ "learning_rate": 9.978714805157398e-05,
834
+ "loss": 0.393,
835
+ "step": 118
836
+ },
837
+ {
838
+ "epoch": 0.17841079460269865,
839
+ "grad_norm": 0.1792721008105896,
840
+ "learning_rate": 9.97796195883804e-05,
841
+ "loss": 0.4153,
842
+ "step": 119
843
+ },
844
+ {
845
+ "epoch": 0.17991004497751126,
846
+ "grad_norm": 0.1707862877011526,
847
+ "learning_rate": 9.97719605839377e-05,
848
+ "loss": 0.4261,
849
+ "step": 120
850
+ },
851
+ {
852
+ "epoch": 0.18140929535232383,
853
+ "grad_norm": 0.22642112261502503,
854
+ "learning_rate": 9.97641710583307e-05,
855
+ "loss": 0.4251,
856
+ "step": 121
857
+ },
858
+ {
859
+ "epoch": 0.18290854572713644,
860
+ "grad_norm": 0.17799712739086393,
861
+ "learning_rate": 9.975625103198656e-05,
862
+ "loss": 0.3727,
863
+ "step": 122
864
+ },
865
+ {
866
+ "epoch": 0.18440779610194902,
867
+ "grad_norm": 0.19076577337705697,
868
+ "learning_rate": 9.974820052567459e-05,
869
+ "loss": 0.4096,
870
+ "step": 123
871
+ },
872
+ {
873
+ "epoch": 0.18590704647676162,
874
+ "grad_norm": 0.1714002069561427,
875
+ "learning_rate": 9.974001956050636e-05,
876
+ "loss": 0.3812,
877
+ "step": 124
878
+ },
879
+ {
880
+ "epoch": 0.1874062968515742,
881
+ "grad_norm": 0.19560455771266627,
882
+ "learning_rate": 9.973170815793543e-05,
883
+ "loss": 0.384,
884
+ "step": 125
885
+ },
886
+ {
887
+ "epoch": 0.1889055472263868,
888
+ "grad_norm": 0.19011043215359408,
889
+ "learning_rate": 9.972326633975752e-05,
890
+ "loss": 0.3454,
891
+ "step": 126
892
+ },
893
+ {
894
+ "epoch": 0.1904047976011994,
895
+ "grad_norm": 0.17831195896279137,
896
+ "learning_rate": 9.971469412811032e-05,
897
+ "loss": 0.3592,
898
+ "step": 127
899
+ },
900
+ {
901
+ "epoch": 0.191904047976012,
902
+ "grad_norm": 0.19815244106934343,
903
+ "learning_rate": 9.970599154547345e-05,
904
+ "loss": 0.4755,
905
+ "step": 128
906
+ },
907
+ {
908
+ "epoch": 0.1934032983508246,
909
+ "grad_norm": 0.20957770702413298,
910
+ "learning_rate": 9.96971586146684e-05,
911
+ "loss": 0.3628,
912
+ "step": 129
913
+ },
914
+ {
915
+ "epoch": 0.19490254872563717,
916
+ "grad_norm": 0.2668865421763697,
917
+ "learning_rate": 9.968819535885851e-05,
918
+ "loss": 0.4102,
919
+ "step": 130
920
+ },
921
+ {
922
+ "epoch": 0.19640179910044978,
923
+ "grad_norm": 0.2090110543133345,
924
+ "learning_rate": 9.967910180154889e-05,
925
+ "loss": 0.4069,
926
+ "step": 131
927
+ },
928
+ {
929
+ "epoch": 0.19790104947526238,
930
+ "grad_norm": 0.24642703407803873,
931
+ "learning_rate": 9.966987796658631e-05,
932
+ "loss": 0.4035,
933
+ "step": 132
934
+ },
935
+ {
936
+ "epoch": 0.19940029985007496,
937
+ "grad_norm": 0.1842758332631964,
938
+ "learning_rate": 9.966052387815923e-05,
939
+ "loss": 0.3659,
940
+ "step": 133
941
+ },
942
+ {
943
+ "epoch": 0.20089955022488756,
944
+ "grad_norm": 0.24323992002330513,
945
+ "learning_rate": 9.965103956079764e-05,
946
+ "loss": 0.3514,
947
+ "step": 134
948
+ },
949
+ {
950
+ "epoch": 0.20239880059970014,
951
+ "grad_norm": 0.1936647477893177,
952
+ "learning_rate": 9.964142503937305e-05,
953
+ "loss": 0.4008,
954
+ "step": 135
955
+ },
956
+ {
957
+ "epoch": 0.20389805097451275,
958
+ "grad_norm": 0.18353917867126332,
959
+ "learning_rate": 9.963168033909842e-05,
960
+ "loss": 0.4653,
961
+ "step": 136
962
+ },
963
+ {
964
+ "epoch": 0.20539730134932535,
965
+ "grad_norm": 0.17984301535007757,
966
+ "learning_rate": 9.962180548552812e-05,
967
+ "loss": 0.3469,
968
+ "step": 137
969
+ },
970
+ {
971
+ "epoch": 0.20689655172413793,
972
+ "grad_norm": 0.15120437502884132,
973
+ "learning_rate": 9.961180050455776e-05,
974
+ "loss": 0.351,
975
+ "step": 138
976
+ },
977
+ {
978
+ "epoch": 0.20839580209895053,
979
+ "grad_norm": 0.1940183704497967,
980
+ "learning_rate": 9.960166542242429e-05,
981
+ "loss": 0.4502,
982
+ "step": 139
983
+ },
984
+ {
985
+ "epoch": 0.2098950524737631,
986
+ "grad_norm": 0.18140590696808803,
987
+ "learning_rate": 9.959140026570571e-05,
988
+ "loss": 0.3881,
989
+ "step": 140
990
+ },
991
+ {
992
+ "epoch": 0.21139430284857572,
993
+ "grad_norm": 0.20509489934011485,
994
+ "learning_rate": 9.958100506132127e-05,
995
+ "loss": 0.3989,
996
+ "step": 141
997
+ },
998
+ {
999
+ "epoch": 0.2128935532233883,
1000
+ "grad_norm": 0.22956081410465642,
1001
+ "learning_rate": 9.957047983653112e-05,
1002
+ "loss": 0.4364,
1003
+ "step": 142
1004
+ },
1005
+ {
1006
+ "epoch": 0.2143928035982009,
1007
+ "grad_norm": 0.22742506736584492,
1008
+ "learning_rate": 9.955982461893648e-05,
1009
+ "loss": 0.3759,
1010
+ "step": 143
1011
+ },
1012
+ {
1013
+ "epoch": 0.2158920539730135,
1014
+ "grad_norm": 0.21527165492782585,
1015
+ "learning_rate": 9.95490394364794e-05,
1016
+ "loss": 0.2983,
1017
+ "step": 144
1018
+ },
1019
+ {
1020
+ "epoch": 0.21739130434782608,
1021
+ "grad_norm": 0.21962083012454633,
1022
+ "learning_rate": 9.953812431744276e-05,
1023
+ "loss": 0.3671,
1024
+ "step": 145
1025
+ },
1026
+ {
1027
+ "epoch": 0.21889055472263869,
1028
+ "grad_norm": 0.23771093531148652,
1029
+ "learning_rate": 9.952707929045018e-05,
1030
+ "loss": 0.4624,
1031
+ "step": 146
1032
+ },
1033
+ {
1034
+ "epoch": 0.22038980509745126,
1035
+ "grad_norm": 0.19791375536680578,
1036
+ "learning_rate": 9.951590438446597e-05,
1037
+ "loss": 0.3225,
1038
+ "step": 147
1039
+ },
1040
+ {
1041
+ "epoch": 0.22188905547226387,
1042
+ "grad_norm": 0.32250544686686966,
1043
+ "learning_rate": 9.950459962879501e-05,
1044
+ "loss": 0.5021,
1045
+ "step": 148
1046
+ },
1047
+ {
1048
+ "epoch": 0.22338830584707647,
1049
+ "grad_norm": 0.19025646517044104,
1050
+ "learning_rate": 9.949316505308271e-05,
1051
+ "loss": 0.3362,
1052
+ "step": 149
1053
+ },
1054
+ {
1055
+ "epoch": 0.22488755622188905,
1056
+ "grad_norm": 0.23171598436518906,
1057
+ "learning_rate": 9.948160068731492e-05,
1058
+ "loss": 0.4147,
1059
+ "step": 150
1060
+ },
1061
+ {
1062
+ "epoch": 0.22638680659670166,
1063
+ "grad_norm": 0.21490934617214158,
1064
+ "learning_rate": 9.946990656181781e-05,
1065
+ "loss": 0.3718,
1066
+ "step": 151
1067
+ },
1068
+ {
1069
+ "epoch": 0.22788605697151423,
1070
+ "grad_norm": 0.1848795090470031,
1071
+ "learning_rate": 9.94580827072579e-05,
1072
+ "loss": 0.276,
1073
+ "step": 152
1074
+ },
1075
+ {
1076
+ "epoch": 0.22938530734632684,
1077
+ "grad_norm": 0.25502312806021005,
1078
+ "learning_rate": 9.944612915464183e-05,
1079
+ "loss": 0.4083,
1080
+ "step": 153
1081
+ },
1082
+ {
1083
+ "epoch": 0.23088455772113944,
1084
+ "grad_norm": 0.20542670291097567,
1085
+ "learning_rate": 9.943404593531642e-05,
1086
+ "loss": 0.3809,
1087
+ "step": 154
1088
+ },
1089
+ {
1090
+ "epoch": 0.23238380809595202,
1091
+ "grad_norm": 0.22026384344552674,
1092
+ "learning_rate": 9.942183308096853e-05,
1093
+ "loss": 0.4016,
1094
+ "step": 155
1095
+ },
1096
+ {
1097
+ "epoch": 0.23388305847076463,
1098
+ "grad_norm": 0.19714327494884368,
1099
+ "learning_rate": 9.940949062362491e-05,
1100
+ "loss": 0.443,
1101
+ "step": 156
1102
+ },
1103
+ {
1104
+ "epoch": 0.2353823088455772,
1105
+ "grad_norm": 0.26398285509147246,
1106
+ "learning_rate": 9.93970185956522e-05,
1107
+ "loss": 0.3443,
1108
+ "step": 157
1109
+ },
1110
+ {
1111
+ "epoch": 0.2368815592203898,
1112
+ "grad_norm": 0.2322940585931519,
1113
+ "learning_rate": 9.938441702975689e-05,
1114
+ "loss": 0.3821,
1115
+ "step": 158
1116
+ },
1117
+ {
1118
+ "epoch": 0.2383808095952024,
1119
+ "grad_norm": 0.29436888057782273,
1120
+ "learning_rate": 9.93716859589851e-05,
1121
+ "loss": 0.4868,
1122
+ "step": 159
1123
+ },
1124
+ {
1125
+ "epoch": 0.239880059970015,
1126
+ "grad_norm": 0.23844652608528716,
1127
+ "learning_rate": 9.935882541672254e-05,
1128
+ "loss": 0.3814,
1129
+ "step": 160
1130
+ },
1131
+ {
1132
+ "epoch": 0.2413793103448276,
1133
+ "grad_norm": 0.23419887056713967,
1134
+ "learning_rate": 9.934583543669453e-05,
1135
+ "loss": 0.3768,
1136
+ "step": 161
1137
+ },
1138
+ {
1139
+ "epoch": 0.24287856071964017,
1140
+ "grad_norm": 0.21730325741302678,
1141
+ "learning_rate": 9.933271605296577e-05,
1142
+ "loss": 0.3537,
1143
+ "step": 162
1144
+ },
1145
+ {
1146
+ "epoch": 0.24437781109445278,
1147
+ "grad_norm": 0.21541731452628599,
1148
+ "learning_rate": 9.931946729994031e-05,
1149
+ "loss": 0.4232,
1150
+ "step": 163
1151
+ },
1152
+ {
1153
+ "epoch": 0.24587706146926536,
1154
+ "grad_norm": 0.23563529017372173,
1155
+ "learning_rate": 9.930608921236144e-05,
1156
+ "loss": 0.3695,
1157
+ "step": 164
1158
+ },
1159
+ {
1160
+ "epoch": 0.24737631184407796,
1161
+ "grad_norm": 0.20973292220708206,
1162
+ "learning_rate": 9.929258182531167e-05,
1163
+ "loss": 0.325,
1164
+ "step": 165
1165
+ },
1166
+ {
1167
+ "epoch": 0.24887556221889057,
1168
+ "grad_norm": 0.22036794347096889,
1169
+ "learning_rate": 9.927894517421252e-05,
1170
+ "loss": 0.368,
1171
+ "step": 166
1172
+ },
1173
+ {
1174
+ "epoch": 0.25037481259370314,
1175
+ "grad_norm": 0.20800248434514584,
1176
+ "learning_rate": 9.926517929482453e-05,
1177
+ "loss": 0.3155,
1178
+ "step": 167
1179
+ },
1180
+ {
1181
+ "epoch": 0.2518740629685157,
1182
+ "grad_norm": 0.23072666850100082,
1183
+ "learning_rate": 9.925128422324711e-05,
1184
+ "loss": 0.3258,
1185
+ "step": 168
1186
+ },
1187
+ {
1188
+ "epoch": 0.25337331334332835,
1189
+ "grad_norm": 0.16981406756976633,
1190
+ "learning_rate": 9.923725999591847e-05,
1191
+ "loss": 0.3147,
1192
+ "step": 169
1193
+ },
1194
+ {
1195
+ "epoch": 0.25487256371814093,
1196
+ "grad_norm": 0.251626688469978,
1197
+ "learning_rate": 9.92231066496155e-05,
1198
+ "loss": 0.3961,
1199
+ "step": 170
1200
+ },
1201
+ {
1202
+ "epoch": 0.2563718140929535,
1203
+ "grad_norm": 0.20921325165098897,
1204
+ "learning_rate": 9.920882422145372e-05,
1205
+ "loss": 0.3919,
1206
+ "step": 171
1207
+ },
1208
+ {
1209
+ "epoch": 0.25787106446776614,
1210
+ "grad_norm": 0.24926718700118658,
1211
+ "learning_rate": 9.919441274888712e-05,
1212
+ "loss": 0.3689,
1213
+ "step": 172
1214
+ },
1215
+ {
1216
+ "epoch": 0.2593703148425787,
1217
+ "grad_norm": 0.23092168312947756,
1218
+ "learning_rate": 9.91798722697081e-05,
1219
+ "loss": 0.3133,
1220
+ "step": 173
1221
+ },
1222
+ {
1223
+ "epoch": 0.2608695652173913,
1224
+ "grad_norm": 0.21351161322704978,
1225
+ "learning_rate": 9.916520282204738e-05,
1226
+ "loss": 0.4149,
1227
+ "step": 174
1228
+ },
1229
+ {
1230
+ "epoch": 0.2623688155922039,
1231
+ "grad_norm": 0.24423924014340928,
1232
+ "learning_rate": 9.915040444437389e-05,
1233
+ "loss": 0.3703,
1234
+ "step": 175
1235
+ },
1236
+ {
1237
+ "epoch": 0.2638680659670165,
1238
+ "grad_norm": 0.29589731176441575,
1239
+ "learning_rate": 9.913547717549463e-05,
1240
+ "loss": 0.4704,
1241
+ "step": 176
1242
+ },
1243
+ {
1244
+ "epoch": 0.2653673163418291,
1245
+ "grad_norm": 0.26638215250105196,
1246
+ "learning_rate": 9.912042105455463e-05,
1247
+ "loss": 0.5577,
1248
+ "step": 177
1249
+ },
1250
+ {
1251
+ "epoch": 0.26686656671664166,
1252
+ "grad_norm": 0.22215452288279844,
1253
+ "learning_rate": 9.91052361210368e-05,
1254
+ "loss": 0.3465,
1255
+ "step": 178
1256
+ },
1257
+ {
1258
+ "epoch": 0.2683658170914543,
1259
+ "grad_norm": 0.17477471131962302,
1260
+ "learning_rate": 9.908992241476188e-05,
1261
+ "loss": 0.2784,
1262
+ "step": 179
1263
+ },
1264
+ {
1265
+ "epoch": 0.2698650674662669,
1266
+ "grad_norm": 0.2780190499956454,
1267
+ "learning_rate": 9.907447997588827e-05,
1268
+ "loss": 0.4407,
1269
+ "step": 180
1270
+ },
1271
+ {
1272
+ "epoch": 0.27136431784107945,
1273
+ "grad_norm": 0.22512347747614664,
1274
+ "learning_rate": 9.905890884491195e-05,
1275
+ "loss": 0.3757,
1276
+ "step": 181
1277
+ },
1278
+ {
1279
+ "epoch": 0.272863568215892,
1280
+ "grad_norm": 0.22917414937417574,
1281
+ "learning_rate": 9.904320906266642e-05,
1282
+ "loss": 0.3356,
1283
+ "step": 182
1284
+ },
1285
+ {
1286
+ "epoch": 0.27436281859070466,
1287
+ "grad_norm": 0.23889689023715455,
1288
+ "learning_rate": 9.902738067032253e-05,
1289
+ "loss": 0.3923,
1290
+ "step": 183
1291
+ },
1292
+ {
1293
+ "epoch": 0.27586206896551724,
1294
+ "grad_norm": 0.25279219700986705,
1295
+ "learning_rate": 9.901142370938837e-05,
1296
+ "loss": 0.3598,
1297
+ "step": 184
1298
+ },
1299
+ {
1300
+ "epoch": 0.2773613193403298,
1301
+ "grad_norm": 0.24373792827720528,
1302
+ "learning_rate": 9.899533822170922e-05,
1303
+ "loss": 0.363,
1304
+ "step": 185
1305
+ },
1306
+ {
1307
+ "epoch": 0.27886056971514245,
1308
+ "grad_norm": 0.21342522741136943,
1309
+ "learning_rate": 9.89791242494674e-05,
1310
+ "loss": 0.3729,
1311
+ "step": 186
1312
+ },
1313
+ {
1314
+ "epoch": 0.280359820089955,
1315
+ "grad_norm": 0.20688482475505948,
1316
+ "learning_rate": 9.896278183518216e-05,
1317
+ "loss": 0.4067,
1318
+ "step": 187
1319
+ },
1320
+ {
1321
+ "epoch": 0.2818590704647676,
1322
+ "grad_norm": 0.238242020004113,
1323
+ "learning_rate": 9.894631102170958e-05,
1324
+ "loss": 0.3519,
1325
+ "step": 188
1326
+ },
1327
+ {
1328
+ "epoch": 0.28335832083958024,
1329
+ "grad_norm": 0.20684969370504117,
1330
+ "learning_rate": 9.892971185224245e-05,
1331
+ "loss": 0.2947,
1332
+ "step": 189
1333
+ },
1334
+ {
1335
+ "epoch": 0.2848575712143928,
1336
+ "grad_norm": 0.24536501295200672,
1337
+ "learning_rate": 9.891298437031014e-05,
1338
+ "loss": 0.4395,
1339
+ "step": 190
1340
+ },
1341
+ {
1342
+ "epoch": 0.2863568215892054,
1343
+ "grad_norm": 0.23631131035836886,
1344
+ "learning_rate": 9.889612861977853e-05,
1345
+ "loss": 0.4022,
1346
+ "step": 191
1347
+ },
1348
+ {
1349
+ "epoch": 0.28785607196401797,
1350
+ "grad_norm": 0.20688289767576518,
1351
+ "learning_rate": 9.887914464484988e-05,
1352
+ "loss": 0.3575,
1353
+ "step": 192
1354
+ },
1355
+ {
1356
+ "epoch": 0.2893553223388306,
1357
+ "grad_norm": 0.23795476789993122,
1358
+ "learning_rate": 9.886203249006265e-05,
1359
+ "loss": 0.488,
1360
+ "step": 193
1361
+ },
1362
+ {
1363
+ "epoch": 0.2908545727136432,
1364
+ "grad_norm": 0.1980321443248457,
1365
+ "learning_rate": 9.884479220029151e-05,
1366
+ "loss": 0.3239,
1367
+ "step": 194
1368
+ },
1369
+ {
1370
+ "epoch": 0.29235382308845576,
1371
+ "grad_norm": 0.2508337482224898,
1372
+ "learning_rate": 9.882742382074707e-05,
1373
+ "loss": 0.3784,
1374
+ "step": 195
1375
+ },
1376
+ {
1377
+ "epoch": 0.2938530734632684,
1378
+ "grad_norm": 0.2705233150633363,
1379
+ "learning_rate": 9.88099273969759e-05,
1380
+ "loss": 0.3126,
1381
+ "step": 196
1382
+ },
1383
+ {
1384
+ "epoch": 0.29535232383808097,
1385
+ "grad_norm": 0.28528624064091823,
1386
+ "learning_rate": 9.879230297486034e-05,
1387
+ "loss": 0.3765,
1388
+ "step": 197
1389
+ },
1390
+ {
1391
+ "epoch": 0.29685157421289354,
1392
+ "grad_norm": 0.27223486197115127,
1393
+ "learning_rate": 9.877455060061838e-05,
1394
+ "loss": 0.3464,
1395
+ "step": 198
1396
+ },
1397
+ {
1398
+ "epoch": 0.2983508245877061,
1399
+ "grad_norm": 0.20975877053780698,
1400
+ "learning_rate": 9.875667032080353e-05,
1401
+ "loss": 0.3696,
1402
+ "step": 199
1403
+ },
1404
+ {
1405
+ "epoch": 0.29985007496251875,
1406
+ "grad_norm": 0.20202593056199547,
1407
+ "learning_rate": 9.873866218230476e-05,
1408
+ "loss": 0.327,
1409
+ "step": 200
1410
+ },
1411
+ {
1412
+ "epoch": 0.29985007496251875,
1413
+ "eval_loss": 0.3760811984539032,
1414
+ "eval_runtime": 9.3971,
1415
+ "eval_samples_per_second": 5.746,
1416
+ "eval_steps_per_second": 1.49,
1417
+ "step": 200
1418
+ },
1419
+ {
1420
+ "epoch": 0.30134932533733133,
1421
+ "grad_norm": 0.20797626168359745,
1422
+ "learning_rate": 9.872052623234632e-05,
1423
+ "loss": 0.2917,
1424
+ "step": 201
1425
+ },
1426
+ {
1427
+ "epoch": 0.3028485757121439,
1428
+ "grad_norm": 0.27008080271764223,
1429
+ "learning_rate": 9.870226251848758e-05,
1430
+ "loss": 0.3787,
1431
+ "step": 202
1432
+ },
1433
+ {
1434
+ "epoch": 0.30434782608695654,
1435
+ "grad_norm": 0.2653659753832694,
1436
+ "learning_rate": 9.868387108862307e-05,
1437
+ "loss": 0.3147,
1438
+ "step": 203
1439
+ },
1440
+ {
1441
+ "epoch": 0.3058470764617691,
1442
+ "grad_norm": 0.22329571999161857,
1443
+ "learning_rate": 9.866535199098212e-05,
1444
+ "loss": 0.2917,
1445
+ "step": 204
1446
+ },
1447
+ {
1448
+ "epoch": 0.3073463268365817,
1449
+ "grad_norm": 0.25966484872877343,
1450
+ "learning_rate": 9.864670527412891e-05,
1451
+ "loss": 0.3668,
1452
+ "step": 205
1453
+ },
1454
+ {
1455
+ "epoch": 0.30884557721139433,
1456
+ "grad_norm": 0.2521720904017885,
1457
+ "learning_rate": 9.86279309869623e-05,
1458
+ "loss": 0.3673,
1459
+ "step": 206
1460
+ },
1461
+ {
1462
+ "epoch": 0.3103448275862069,
1463
+ "grad_norm": 0.23732244427926463,
1464
+ "learning_rate": 9.860902917871567e-05,
1465
+ "loss": 0.3677,
1466
+ "step": 207
1467
+ },
1468
+ {
1469
+ "epoch": 0.3118440779610195,
1470
+ "grad_norm": 0.2955078675425853,
1471
+ "learning_rate": 9.858999989895678e-05,
1472
+ "loss": 0.4393,
1473
+ "step": 208
1474
+ },
1475
+ {
1476
+ "epoch": 0.31334332833583206,
1477
+ "grad_norm": 0.22332660189204384,
1478
+ "learning_rate": 9.857084319758772e-05,
1479
+ "loss": 0.3689,
1480
+ "step": 209
1481
+ },
1482
+ {
1483
+ "epoch": 0.3148425787106447,
1484
+ "grad_norm": 0.22760182192880996,
1485
+ "learning_rate": 9.85515591248447e-05,
1486
+ "loss": 0.3867,
1487
+ "step": 210
1488
+ },
1489
+ {
1490
+ "epoch": 0.31634182908545727,
1491
+ "grad_norm": 0.24914198184951847,
1492
+ "learning_rate": 9.853214773129796e-05,
1493
+ "loss": 0.3589,
1494
+ "step": 211
1495
+ },
1496
+ {
1497
+ "epoch": 0.31784107946026985,
1498
+ "grad_norm": 0.23576174897868454,
1499
+ "learning_rate": 9.851260906785161e-05,
1500
+ "loss": 0.3448,
1501
+ "step": 212
1502
+ },
1503
+ {
1504
+ "epoch": 0.3193403298350825,
1505
+ "grad_norm": 0.29164953095585383,
1506
+ "learning_rate": 9.849294318574351e-05,
1507
+ "loss": 0.4416,
1508
+ "step": 213
1509
+ },
1510
+ {
1511
+ "epoch": 0.32083958020989506,
1512
+ "grad_norm": 0.2586079180874359,
1513
+ "learning_rate": 9.847315013654517e-05,
1514
+ "loss": 0.368,
1515
+ "step": 214
1516
+ },
1517
+ {
1518
+ "epoch": 0.32233883058470764,
1519
+ "grad_norm": 0.2339697800825202,
1520
+ "learning_rate": 9.845322997216153e-05,
1521
+ "loss": 0.3323,
1522
+ "step": 215
1523
+ },
1524
+ {
1525
+ "epoch": 0.3238380809595202,
1526
+ "grad_norm": 0.23665725114029154,
1527
+ "learning_rate": 9.843318274483089e-05,
1528
+ "loss": 0.389,
1529
+ "step": 216
1530
+ },
1531
+ {
1532
+ "epoch": 0.32533733133433285,
1533
+ "grad_norm": 0.20956611594099658,
1534
+ "learning_rate": 9.84130085071248e-05,
1535
+ "loss": 0.3477,
1536
+ "step": 217
1537
+ },
1538
+ {
1539
+ "epoch": 0.3268365817091454,
1540
+ "grad_norm": 0.26392168214587325,
1541
+ "learning_rate": 9.839270731194781e-05,
1542
+ "loss": 0.3638,
1543
+ "step": 218
1544
+ },
1545
+ {
1546
+ "epoch": 0.328335832083958,
1547
+ "grad_norm": 0.22253773776911556,
1548
+ "learning_rate": 9.837227921253746e-05,
1549
+ "loss": 0.3204,
1550
+ "step": 219
1551
+ },
1552
+ {
1553
+ "epoch": 0.32983508245877063,
1554
+ "grad_norm": 0.2665743634558936,
1555
+ "learning_rate": 9.835172426246406e-05,
1556
+ "loss": 0.3809,
1557
+ "step": 220
1558
+ },
1559
+ {
1560
+ "epoch": 0.3313343328335832,
1561
+ "grad_norm": 0.24556387681728048,
1562
+ "learning_rate": 9.833104251563056e-05,
1563
+ "loss": 0.3658,
1564
+ "step": 221
1565
+ },
1566
+ {
1567
+ "epoch": 0.3328335832083958,
1568
+ "grad_norm": 0.24141132534691834,
1569
+ "learning_rate": 9.831023402627244e-05,
1570
+ "loss": 0.3878,
1571
+ "step": 222
1572
+ },
1573
+ {
1574
+ "epoch": 0.3343328335832084,
1575
+ "grad_norm": 0.21299779644052697,
1576
+ "learning_rate": 9.828929884895752e-05,
1577
+ "loss": 0.3385,
1578
+ "step": 223
1579
+ },
1580
+ {
1581
+ "epoch": 0.335832083958021,
1582
+ "grad_norm": 0.2817444763775132,
1583
+ "learning_rate": 9.826823703858589e-05,
1584
+ "loss": 0.3609,
1585
+ "step": 224
1586
+ },
1587
+ {
1588
+ "epoch": 0.3373313343328336,
1589
+ "grad_norm": 0.23675212339499258,
1590
+ "learning_rate": 9.824704865038968e-05,
1591
+ "loss": 0.345,
1592
+ "step": 225
1593
+ },
1594
+ {
1595
+ "epoch": 0.33883058470764615,
1596
+ "grad_norm": 0.26616531581298974,
1597
+ "learning_rate": 9.822573373993295e-05,
1598
+ "loss": 0.4403,
1599
+ "step": 226
1600
+ },
1601
+ {
1602
+ "epoch": 0.3403298350824588,
1603
+ "grad_norm": 0.25547096640384365,
1604
+ "learning_rate": 9.820429236311158e-05,
1605
+ "loss": 0.329,
1606
+ "step": 227
1607
+ },
1608
+ {
1609
+ "epoch": 0.34182908545727136,
1610
+ "grad_norm": 0.2069046648269492,
1611
+ "learning_rate": 9.81827245761531e-05,
1612
+ "loss": 0.3093,
1613
+ "step": 228
1614
+ },
1615
+ {
1616
+ "epoch": 0.34332833583208394,
1617
+ "grad_norm": 0.22028926871844512,
1618
+ "learning_rate": 9.816103043561648e-05,
1619
+ "loss": 0.3145,
1620
+ "step": 229
1621
+ },
1622
+ {
1623
+ "epoch": 0.3448275862068966,
1624
+ "grad_norm": 0.22267908189639102,
1625
+ "learning_rate": 9.81392099983921e-05,
1626
+ "loss": 0.3522,
1627
+ "step": 230
1628
+ },
1629
+ {
1630
+ "epoch": 0.34632683658170915,
1631
+ "grad_norm": 0.26736838753831266,
1632
+ "learning_rate": 9.811726332170153e-05,
1633
+ "loss": 0.3131,
1634
+ "step": 231
1635
+ },
1636
+ {
1637
+ "epoch": 0.34782608695652173,
1638
+ "grad_norm": 0.27031685049924925,
1639
+ "learning_rate": 9.809519046309732e-05,
1640
+ "loss": 0.3304,
1641
+ "step": 232
1642
+ },
1643
+ {
1644
+ "epoch": 0.3493253373313343,
1645
+ "grad_norm": 0.2287552620096828,
1646
+ "learning_rate": 9.8072991480463e-05,
1647
+ "loss": 0.3362,
1648
+ "step": 233
1649
+ },
1650
+ {
1651
+ "epoch": 0.35082458770614694,
1652
+ "grad_norm": 0.248714164280034,
1653
+ "learning_rate": 9.805066643201282e-05,
1654
+ "loss": 0.4216,
1655
+ "step": 234
1656
+ },
1657
+ {
1658
+ "epoch": 0.3523238380809595,
1659
+ "grad_norm": 0.2602266675933893,
1660
+ "learning_rate": 9.802821537629161e-05,
1661
+ "loss": 0.3405,
1662
+ "step": 235
1663
+ },
1664
+ {
1665
+ "epoch": 0.3538230884557721,
1666
+ "grad_norm": 0.2605825891602772,
1667
+ "learning_rate": 9.800563837217464e-05,
1668
+ "loss": 0.352,
1669
+ "step": 236
1670
+ },
1671
+ {
1672
+ "epoch": 0.3553223388305847,
1673
+ "grad_norm": 0.25964426129393503,
1674
+ "learning_rate": 9.798293547886748e-05,
1675
+ "loss": 0.3862,
1676
+ "step": 237
1677
+ },
1678
+ {
1679
+ "epoch": 0.3568215892053973,
1680
+ "grad_norm": 0.2700571209610955,
1681
+ "learning_rate": 9.796010675590581e-05,
1682
+ "loss": 0.3916,
1683
+ "step": 238
1684
+ },
1685
+ {
1686
+ "epoch": 0.3583208395802099,
1687
+ "grad_norm": 0.26143094542633033,
1688
+ "learning_rate": 9.79371522631553e-05,
1689
+ "loss": 0.4112,
1690
+ "step": 239
1691
+ },
1692
+ {
1693
+ "epoch": 0.3598200899550225,
1694
+ "grad_norm": 0.3029423068355501,
1695
+ "learning_rate": 9.791407206081144e-05,
1696
+ "loss": 0.3856,
1697
+ "step": 240
1698
+ },
1699
+ {
1700
+ "epoch": 0.3613193403298351,
1701
+ "grad_norm": 0.253427922964557,
1702
+ "learning_rate": 9.789086620939936e-05,
1703
+ "loss": 0.3722,
1704
+ "step": 241
1705
+ },
1706
+ {
1707
+ "epoch": 0.36281859070464767,
1708
+ "grad_norm": 0.3183096353136163,
1709
+ "learning_rate": 9.78675347697737e-05,
1710
+ "loss": 0.3626,
1711
+ "step": 242
1712
+ },
1713
+ {
1714
+ "epoch": 0.36431784107946025,
1715
+ "grad_norm": 0.2947544176016858,
1716
+ "learning_rate": 9.784407780311845e-05,
1717
+ "loss": 0.381,
1718
+ "step": 243
1719
+ },
1720
+ {
1721
+ "epoch": 0.3658170914542729,
1722
+ "grad_norm": 0.2729444657167005,
1723
+ "learning_rate": 9.782049537094675e-05,
1724
+ "loss": 0.3377,
1725
+ "step": 244
1726
+ },
1727
+ {
1728
+ "epoch": 0.36731634182908546,
1729
+ "grad_norm": 0.23178174073132465,
1730
+ "learning_rate": 9.779678753510081e-05,
1731
+ "loss": 0.3414,
1732
+ "step": 245
1733
+ },
1734
+ {
1735
+ "epoch": 0.36881559220389803,
1736
+ "grad_norm": 0.2825828472645432,
1737
+ "learning_rate": 9.777295435775165e-05,
1738
+ "loss": 0.3497,
1739
+ "step": 246
1740
+ },
1741
+ {
1742
+ "epoch": 0.37031484257871067,
1743
+ "grad_norm": 0.26645833461164664,
1744
+ "learning_rate": 9.774899590139896e-05,
1745
+ "loss": 0.4364,
1746
+ "step": 247
1747
+ },
1748
+ {
1749
+ "epoch": 0.37181409295352325,
1750
+ "grad_norm": 0.23712519611329114,
1751
+ "learning_rate": 9.772491222887108e-05,
1752
+ "loss": 0.3539,
1753
+ "step": 248
1754
+ },
1755
+ {
1756
+ "epoch": 0.3733133433283358,
1757
+ "grad_norm": 0.2733836951147141,
1758
+ "learning_rate": 9.770070340332456e-05,
1759
+ "loss": 0.3737,
1760
+ "step": 249
1761
+ },
1762
+ {
1763
+ "epoch": 0.3748125937031484,
1764
+ "grad_norm": 0.2878369504999718,
1765
+ "learning_rate": 9.767636948824429e-05,
1766
+ "loss": 0.3587,
1767
+ "step": 250
1768
+ },
1769
+ {
1770
+ "epoch": 0.37631184407796103,
1771
+ "grad_norm": 0.23506650877109067,
1772
+ "learning_rate": 9.765191054744305e-05,
1773
+ "loss": 0.3227,
1774
+ "step": 251
1775
+ },
1776
+ {
1777
+ "epoch": 0.3778110944527736,
1778
+ "grad_norm": 0.2289756510820255,
1779
+ "learning_rate": 9.762732664506162e-05,
1780
+ "loss": 0.3295,
1781
+ "step": 252
1782
+ },
1783
+ {
1784
+ "epoch": 0.3793103448275862,
1785
+ "grad_norm": 0.25655839600121205,
1786
+ "learning_rate": 9.760261784556839e-05,
1787
+ "loss": 0.4075,
1788
+ "step": 253
1789
+ },
1790
+ {
1791
+ "epoch": 0.3808095952023988,
1792
+ "grad_norm": 0.2828449228240892,
1793
+ "learning_rate": 9.757778421375931e-05,
1794
+ "loss": 0.3603,
1795
+ "step": 254
1796
+ },
1797
+ {
1798
+ "epoch": 0.3823088455772114,
1799
+ "grad_norm": 0.2264172708698365,
1800
+ "learning_rate": 9.755282581475769e-05,
1801
+ "loss": 0.3591,
1802
+ "step": 255
1803
+ },
1804
+ {
1805
+ "epoch": 0.383808095952024,
1806
+ "grad_norm": 0.32419712905550324,
1807
+ "learning_rate": 9.752774271401402e-05,
1808
+ "loss": 0.3927,
1809
+ "step": 256
1810
+ },
1811
+ {
1812
+ "epoch": 0.3853073463268366,
1813
+ "grad_norm": 0.21261632252005214,
1814
+ "learning_rate": 9.75025349773058e-05,
1815
+ "loss": 0.337,
1816
+ "step": 257
1817
+ },
1818
+ {
1819
+ "epoch": 0.3868065967016492,
1820
+ "grad_norm": 0.24672564693562782,
1821
+ "learning_rate": 9.747720267073739e-05,
1822
+ "loss": 0.3773,
1823
+ "step": 258
1824
+ },
1825
+ {
1826
+ "epoch": 0.38830584707646176,
1827
+ "grad_norm": 0.3289152739555464,
1828
+ "learning_rate": 9.745174586073981e-05,
1829
+ "loss": 0.3636,
1830
+ "step": 259
1831
+ },
1832
+ {
1833
+ "epoch": 0.38980509745127434,
1834
+ "grad_norm": 0.23791401611376534,
1835
+ "learning_rate": 9.742616461407058e-05,
1836
+ "loss": 0.3716,
1837
+ "step": 260
1838
+ },
1839
+ {
1840
+ "epoch": 0.391304347826087,
1841
+ "grad_norm": 0.3244410680415994,
1842
+ "learning_rate": 9.740045899781352e-05,
1843
+ "loss": 0.4267,
1844
+ "step": 261
1845
+ },
1846
+ {
1847
+ "epoch": 0.39280359820089955,
1848
+ "grad_norm": 0.2832504337710305,
1849
+ "learning_rate": 9.737462907937864e-05,
1850
+ "loss": 0.4039,
1851
+ "step": 262
1852
+ },
1853
+ {
1854
+ "epoch": 0.39430284857571213,
1855
+ "grad_norm": 0.26259910556582383,
1856
+ "learning_rate": 9.734867492650186e-05,
1857
+ "loss": 0.2972,
1858
+ "step": 263
1859
+ },
1860
+ {
1861
+ "epoch": 0.39580209895052476,
1862
+ "grad_norm": 0.30388172270525105,
1863
+ "learning_rate": 9.732259660724494e-05,
1864
+ "loss": 0.4006,
1865
+ "step": 264
1866
+ },
1867
+ {
1868
+ "epoch": 0.39730134932533734,
1869
+ "grad_norm": 0.30010730154486515,
1870
+ "learning_rate": 9.729639418999523e-05,
1871
+ "loss": 0.3727,
1872
+ "step": 265
1873
+ },
1874
+ {
1875
+ "epoch": 0.3988005997001499,
1876
+ "grad_norm": 0.2571533839616395,
1877
+ "learning_rate": 9.727006774346551e-05,
1878
+ "loss": 0.4284,
1879
+ "step": 266
1880
+ },
1881
+ {
1882
+ "epoch": 0.4002998500749625,
1883
+ "grad_norm": 0.24941510856749405,
1884
+ "learning_rate": 9.724361733669381e-05,
1885
+ "loss": 0.3231,
1886
+ "step": 267
1887
+ },
1888
+ {
1889
+ "epoch": 0.4017991004497751,
1890
+ "grad_norm": 0.2594599918439636,
1891
+ "learning_rate": 9.721704303904325e-05,
1892
+ "loss": 0.3587,
1893
+ "step": 268
1894
+ },
1895
+ {
1896
+ "epoch": 0.4032983508245877,
1897
+ "grad_norm": 0.21929175486166685,
1898
+ "learning_rate": 9.719034492020183e-05,
1899
+ "loss": 0.3131,
1900
+ "step": 269
1901
+ },
1902
+ {
1903
+ "epoch": 0.4047976011994003,
1904
+ "grad_norm": 0.2542813579445722,
1905
+ "learning_rate": 9.716352305018223e-05,
1906
+ "loss": 0.3283,
1907
+ "step": 270
1908
+ },
1909
+ {
1910
+ "epoch": 0.4062968515742129,
1911
+ "grad_norm": 0.2327355156412815,
1912
+ "learning_rate": 9.713657749932172e-05,
1913
+ "loss": 0.31,
1914
+ "step": 271
1915
+ },
1916
+ {
1917
+ "epoch": 0.4077961019490255,
1918
+ "grad_norm": 0.2921037233648013,
1919
+ "learning_rate": 9.710950833828182e-05,
1920
+ "loss": 0.3507,
1921
+ "step": 272
1922
+ },
1923
+ {
1924
+ "epoch": 0.40929535232383807,
1925
+ "grad_norm": 0.27754577604624747,
1926
+ "learning_rate": 9.708231563804828e-05,
1927
+ "loss": 0.4316,
1928
+ "step": 273
1929
+ },
1930
+ {
1931
+ "epoch": 0.4107946026986507,
1932
+ "grad_norm": 0.2810106040701159,
1933
+ "learning_rate": 9.705499946993078e-05,
1934
+ "loss": 0.3343,
1935
+ "step": 274
1936
+ },
1937
+ {
1938
+ "epoch": 0.4122938530734633,
1939
+ "grad_norm": 0.3081928955592783,
1940
+ "learning_rate": 9.702755990556275e-05,
1941
+ "loss": 0.3522,
1942
+ "step": 275
1943
+ },
1944
+ {
1945
+ "epoch": 0.41379310344827586,
1946
+ "grad_norm": 0.31027462279081486,
1947
+ "learning_rate": 9.699999701690133e-05,
1948
+ "loss": 0.3991,
1949
+ "step": 276
1950
+ },
1951
+ {
1952
+ "epoch": 0.41529235382308843,
1953
+ "grad_norm": 0.30769321809153244,
1954
+ "learning_rate": 9.697231087622691e-05,
1955
+ "loss": 0.4251,
1956
+ "step": 277
1957
+ },
1958
+ {
1959
+ "epoch": 0.41679160419790107,
1960
+ "grad_norm": 0.22008049482711858,
1961
+ "learning_rate": 9.694450155614319e-05,
1962
+ "loss": 0.274,
1963
+ "step": 278
1964
+ },
1965
+ {
1966
+ "epoch": 0.41829085457271364,
1967
+ "grad_norm": 0.3051942730354458,
1968
+ "learning_rate": 9.691656912957685e-05,
1969
+ "loss": 0.3935,
1970
+ "step": 279
1971
+ },
1972
+ {
1973
+ "epoch": 0.4197901049475262,
1974
+ "grad_norm": 0.2494353947223113,
1975
+ "learning_rate": 9.688851366977747e-05,
1976
+ "loss": 0.3325,
1977
+ "step": 280
1978
+ },
1979
+ {
1980
+ "epoch": 0.42128935532233885,
1981
+ "grad_norm": 0.28093897530314915,
1982
+ "learning_rate": 9.686033525031719e-05,
1983
+ "loss": 0.3216,
1984
+ "step": 281
1985
+ },
1986
+ {
1987
+ "epoch": 0.42278860569715143,
1988
+ "grad_norm": 0.20485653911930882,
1989
+ "learning_rate": 9.683203394509063e-05,
1990
+ "loss": 0.3121,
1991
+ "step": 282
1992
+ },
1993
+ {
1994
+ "epoch": 0.424287856071964,
1995
+ "grad_norm": 0.2524923468239394,
1996
+ "learning_rate": 9.680360982831466e-05,
1997
+ "loss": 0.3625,
1998
+ "step": 283
1999
+ },
2000
+ {
2001
+ "epoch": 0.4257871064467766,
2002
+ "grad_norm": 0.26133961109245535,
2003
+ "learning_rate": 9.677506297452823e-05,
2004
+ "loss": 0.3233,
2005
+ "step": 284
2006
+ },
2007
+ {
2008
+ "epoch": 0.4272863568215892,
2009
+ "grad_norm": 0.22941502055388824,
2010
+ "learning_rate": 9.674639345859214e-05,
2011
+ "loss": 0.3371,
2012
+ "step": 285
2013
+ },
2014
+ {
2015
+ "epoch": 0.4287856071964018,
2016
+ "grad_norm": 0.2280161291194043,
2017
+ "learning_rate": 9.671760135568881e-05,
2018
+ "loss": 0.322,
2019
+ "step": 286
2020
+ },
2021
+ {
2022
+ "epoch": 0.4302848575712144,
2023
+ "grad_norm": 0.2611909815910577,
2024
+ "learning_rate": 9.668868674132224e-05,
2025
+ "loss": 0.3443,
2026
+ "step": 287
2027
+ },
2028
+ {
2029
+ "epoch": 0.431784107946027,
2030
+ "grad_norm": 0.2420410455674395,
2031
+ "learning_rate": 9.665964969131757e-05,
2032
+ "loss": 0.3212,
2033
+ "step": 288
2034
+ },
2035
+ {
2036
+ "epoch": 0.4332833583208396,
2037
+ "grad_norm": 0.21845173026055942,
2038
+ "learning_rate": 9.663049028182111e-05,
2039
+ "loss": 0.3937,
2040
+ "step": 289
2041
+ },
2042
+ {
2043
+ "epoch": 0.43478260869565216,
2044
+ "grad_norm": 0.2729587145211045,
2045
+ "learning_rate": 9.660120858930003e-05,
2046
+ "loss": 0.3894,
2047
+ "step": 290
2048
+ },
2049
+ {
2050
+ "epoch": 0.4362818590704648,
2051
+ "grad_norm": 0.29514998643897117,
2052
+ "learning_rate": 9.657180469054213e-05,
2053
+ "loss": 0.3725,
2054
+ "step": 291
2055
+ },
2056
+ {
2057
+ "epoch": 0.43778110944527737,
2058
+ "grad_norm": 0.3635648533154266,
2059
+ "learning_rate": 9.654227866265569e-05,
2060
+ "loss": 0.3328,
2061
+ "step": 292
2062
+ },
2063
+ {
2064
+ "epoch": 0.43928035982008995,
2065
+ "grad_norm": 0.26134711305689295,
2066
+ "learning_rate": 9.651263058306932e-05,
2067
+ "loss": 0.3759,
2068
+ "step": 293
2069
+ },
2070
+ {
2071
+ "epoch": 0.4407796101949025,
2072
+ "grad_norm": 0.2428671238353619,
2073
+ "learning_rate": 9.648286052953161e-05,
2074
+ "loss": 0.2611,
2075
+ "step": 294
2076
+ },
2077
+ {
2078
+ "epoch": 0.44227886056971516,
2079
+ "grad_norm": 0.2307583586352917,
2080
+ "learning_rate": 9.645296858011109e-05,
2081
+ "loss": 0.3202,
2082
+ "step": 295
2083
+ },
2084
+ {
2085
+ "epoch": 0.44377811094452774,
2086
+ "grad_norm": 0.27390329996301765,
2087
+ "learning_rate": 9.642295481319588e-05,
2088
+ "loss": 0.3401,
2089
+ "step": 296
2090
+ },
2091
+ {
2092
+ "epoch": 0.4452773613193403,
2093
+ "grad_norm": 0.3190025822824932,
2094
+ "learning_rate": 9.639281930749362e-05,
2095
+ "loss": 0.4227,
2096
+ "step": 297
2097
+ },
2098
+ {
2099
+ "epoch": 0.44677661169415295,
2100
+ "grad_norm": 0.2598883344131087,
2101
+ "learning_rate": 9.636256214203115e-05,
2102
+ "loss": 0.2825,
2103
+ "step": 298
2104
+ },
2105
+ {
2106
+ "epoch": 0.4482758620689655,
2107
+ "grad_norm": 0.2706626760124441,
2108
+ "learning_rate": 9.633218339615433e-05,
2109
+ "loss": 0.3393,
2110
+ "step": 299
2111
+ },
2112
+ {
2113
+ "epoch": 0.4497751124437781,
2114
+ "grad_norm": 0.24831760559926722,
2115
+ "learning_rate": 9.63016831495279e-05,
2116
+ "loss": 0.2696,
2117
+ "step": 300
2118
+ },
2119
+ {
2120
+ "epoch": 0.4512743628185907,
2121
+ "grad_norm": 0.2440264653259474,
2122
+ "learning_rate": 9.627106148213522e-05,
2123
+ "loss": 0.3319,
2124
+ "step": 301
2125
+ },
2126
+ {
2127
+ "epoch": 0.4527736131934033,
2128
+ "grad_norm": 0.26598694710365517,
2129
+ "learning_rate": 9.624031847427801e-05,
2130
+ "loss": 0.3565,
2131
+ "step": 302
2132
+ },
2133
+ {
2134
+ "epoch": 0.4542728635682159,
2135
+ "grad_norm": 0.24931600850650895,
2136
+ "learning_rate": 9.620945420657624e-05,
2137
+ "loss": 0.3273,
2138
+ "step": 303
2139
+ },
2140
+ {
2141
+ "epoch": 0.45577211394302847,
2142
+ "grad_norm": 0.2338286567685855,
2143
+ "learning_rate": 9.617846875996785e-05,
2144
+ "loss": 0.3654,
2145
+ "step": 304
2146
+ },
2147
+ {
2148
+ "epoch": 0.4572713643178411,
2149
+ "grad_norm": 0.2575110981356596,
2150
+ "learning_rate": 9.61473622157086e-05,
2151
+ "loss": 0.3288,
2152
+ "step": 305
2153
+ },
2154
+ {
2155
+ "epoch": 0.4587706146926537,
2156
+ "grad_norm": 0.28935790088696406,
2157
+ "learning_rate": 9.61161346553717e-05,
2158
+ "loss": 0.3247,
2159
+ "step": 306
2160
+ },
2161
+ {
2162
+ "epoch": 0.46026986506746626,
2163
+ "grad_norm": 0.2774321646396522,
2164
+ "learning_rate": 9.608478616084784e-05,
2165
+ "loss": 0.3503,
2166
+ "step": 307
2167
+ },
2168
+ {
2169
+ "epoch": 0.4617691154422789,
2170
+ "grad_norm": 0.2413752022515758,
2171
+ "learning_rate": 9.605331681434477e-05,
2172
+ "loss": 0.2821,
2173
+ "step": 308
2174
+ },
2175
+ {
2176
+ "epoch": 0.46326836581709147,
2177
+ "grad_norm": 0.28191456971423773,
2178
+ "learning_rate": 9.602172669838721e-05,
2179
+ "loss": 0.3564,
2180
+ "step": 309
2181
+ },
2182
+ {
2183
+ "epoch": 0.46476761619190404,
2184
+ "grad_norm": 0.284058411770928,
2185
+ "learning_rate": 9.599001589581653e-05,
2186
+ "loss": 0.3011,
2187
+ "step": 310
2188
+ },
2189
+ {
2190
+ "epoch": 0.4662668665667166,
2191
+ "grad_norm": 0.22117702063027717,
2192
+ "learning_rate": 9.595818448979061e-05,
2193
+ "loss": 0.2999,
2194
+ "step": 311
2195
+ },
2196
+ {
2197
+ "epoch": 0.46776611694152925,
2198
+ "grad_norm": 0.29800109798421304,
2199
+ "learning_rate": 9.592623256378364e-05,
2200
+ "loss": 0.4059,
2201
+ "step": 312
2202
+ },
2203
+ {
2204
+ "epoch": 0.46926536731634183,
2205
+ "grad_norm": 0.2615376129504821,
2206
+ "learning_rate": 9.589416020158578e-05,
2207
+ "loss": 0.3139,
2208
+ "step": 313
2209
+ },
2210
+ {
2211
+ "epoch": 0.4707646176911544,
2212
+ "grad_norm": 0.2936529105437823,
2213
+ "learning_rate": 9.586196748730305e-05,
2214
+ "loss": 0.3251,
2215
+ "step": 314
2216
+ },
2217
+ {
2218
+ "epoch": 0.47226386806596704,
2219
+ "grad_norm": 0.266468483705676,
2220
+ "learning_rate": 9.582965450535715e-05,
2221
+ "loss": 0.4037,
2222
+ "step": 315
2223
+ },
2224
+ {
2225
+ "epoch": 0.4737631184407796,
2226
+ "grad_norm": 0.24230764465235774,
2227
+ "learning_rate": 9.579722134048506e-05,
2228
+ "loss": 0.3296,
2229
+ "step": 316
2230
+ },
2231
+ {
2232
+ "epoch": 0.4752623688155922,
2233
+ "grad_norm": 0.340728401615647,
2234
+ "learning_rate": 9.576466807773899e-05,
2235
+ "loss": 0.3957,
2236
+ "step": 317
2237
+ },
2238
+ {
2239
+ "epoch": 0.4767616191904048,
2240
+ "grad_norm": 0.2349979259521195,
2241
+ "learning_rate": 9.573199480248606e-05,
2242
+ "loss": 0.2905,
2243
+ "step": 318
2244
+ },
2245
+ {
2246
+ "epoch": 0.4782608695652174,
2247
+ "grad_norm": 0.2645057636612975,
2248
+ "learning_rate": 9.569920160040815e-05,
2249
+ "loss": 0.3211,
2250
+ "step": 319
2251
+ },
2252
+ {
2253
+ "epoch": 0.47976011994003,
2254
+ "grad_norm": 0.3352556160783962,
2255
+ "learning_rate": 9.56662885575016e-05,
2256
+ "loss": 0.3847,
2257
+ "step": 320
2258
+ },
2259
+ {
2260
+ "epoch": 0.48125937031484256,
2261
+ "grad_norm": 0.26527957662237917,
2262
+ "learning_rate": 9.563325576007701e-05,
2263
+ "loss": 0.3904,
2264
+ "step": 321
2265
+ },
2266
+ {
2267
+ "epoch": 0.4827586206896552,
2268
+ "grad_norm": 0.31309830682681533,
2269
+ "learning_rate": 9.560010329475906e-05,
2270
+ "loss": 0.3364,
2271
+ "step": 322
2272
+ },
2273
+ {
2274
+ "epoch": 0.48425787106446777,
2275
+ "grad_norm": 0.2710524175789773,
2276
+ "learning_rate": 9.556683124848625e-05,
2277
+ "loss": 0.3407,
2278
+ "step": 323
2279
+ },
2280
+ {
2281
+ "epoch": 0.48575712143928035,
2282
+ "grad_norm": 0.24537581603658395,
2283
+ "learning_rate": 9.553343970851059e-05,
2284
+ "loss": 0.2852,
2285
+ "step": 324
2286
+ },
2287
+ {
2288
+ "epoch": 0.487256371814093,
2289
+ "grad_norm": 0.32677142363843803,
2290
+ "learning_rate": 9.549992876239753e-05,
2291
+ "loss": 0.3471,
2292
+ "step": 325
2293
+ },
2294
+ {
2295
+ "epoch": 0.48875562218890556,
2296
+ "grad_norm": 0.2556056612640656,
2297
+ "learning_rate": 9.546629849802562e-05,
2298
+ "loss": 0.2887,
2299
+ "step": 326
2300
+ },
2301
+ {
2302
+ "epoch": 0.49025487256371814,
2303
+ "grad_norm": 0.29207206322173274,
2304
+ "learning_rate": 9.543254900358629e-05,
2305
+ "loss": 0.4087,
2306
+ "step": 327
2307
+ },
2308
+ {
2309
+ "epoch": 0.4917541229385307,
2310
+ "grad_norm": 0.2775712015290828,
2311
+ "learning_rate": 9.539868036758368e-05,
2312
+ "loss": 0.3968,
2313
+ "step": 328
2314
+ },
2315
+ {
2316
+ "epoch": 0.49325337331334335,
2317
+ "grad_norm": 0.26600103017143895,
2318
+ "learning_rate": 9.536469267883433e-05,
2319
+ "loss": 0.3577,
2320
+ "step": 329
2321
+ },
2322
+ {
2323
+ "epoch": 0.4947526236881559,
2324
+ "grad_norm": 0.2701688757517727,
2325
+ "learning_rate": 9.533058602646696e-05,
2326
+ "loss": 0.3859,
2327
+ "step": 330
2328
+ },
2329
+ {
2330
+ "epoch": 0.4962518740629685,
2331
+ "grad_norm": 0.2788155427678552,
2332
+ "learning_rate": 9.529636049992234e-05,
2333
+ "loss": 0.3405,
2334
+ "step": 331
2335
+ },
2336
+ {
2337
+ "epoch": 0.49775112443778113,
2338
+ "grad_norm": 0.2781766465181338,
2339
+ "learning_rate": 9.526201618895291e-05,
2340
+ "loss": 0.3477,
2341
+ "step": 332
2342
+ },
2343
+ {
2344
+ "epoch": 0.4992503748125937,
2345
+ "grad_norm": 0.2464135283146033,
2346
+ "learning_rate": 9.52275531836226e-05,
2347
+ "loss": 0.32,
2348
+ "step": 333
2349
+ },
2350
+ {
2351
+ "epoch": 0.5007496251874063,
2352
+ "grad_norm": 0.3056198636651611,
2353
+ "learning_rate": 9.519297157430665e-05,
2354
+ "loss": 0.3278,
2355
+ "step": 334
2356
+ },
2357
+ {
2358
+ "epoch": 0.5022488755622189,
2359
+ "grad_norm": 0.25415158306661245,
2360
+ "learning_rate": 9.515827145169127e-05,
2361
+ "loss": 0.3504,
2362
+ "step": 335
2363
+ },
2364
+ {
2365
+ "epoch": 0.5037481259370314,
2366
+ "grad_norm": 0.2603363481884891,
2367
+ "learning_rate": 9.51234529067735e-05,
2368
+ "loss": 0.3473,
2369
+ "step": 336
2370
+ },
2371
+ {
2372
+ "epoch": 0.5052473763118441,
2373
+ "grad_norm": 0.30403967173063684,
2374
+ "learning_rate": 9.508851603086093e-05,
2375
+ "loss": 0.4433,
2376
+ "step": 337
2377
+ },
2378
+ {
2379
+ "epoch": 0.5067466266866567,
2380
+ "grad_norm": 0.26939763348535367,
2381
+ "learning_rate": 9.505346091557143e-05,
2382
+ "loss": 0.3103,
2383
+ "step": 338
2384
+ },
2385
+ {
2386
+ "epoch": 0.5082458770614693,
2387
+ "grad_norm": 0.267724719821623,
2388
+ "learning_rate": 9.501828765283295e-05,
2389
+ "loss": 0.3022,
2390
+ "step": 339
2391
+ },
2392
+ {
2393
+ "epoch": 0.5097451274362819,
2394
+ "grad_norm": 0.3407824430683693,
2395
+ "learning_rate": 9.498299633488328e-05,
2396
+ "loss": 0.3032,
2397
+ "step": 340
2398
+ },
2399
+ {
2400
+ "epoch": 0.5112443778110944,
2401
+ "grad_norm": 0.3104505713027564,
2402
+ "learning_rate": 9.494758705426978e-05,
2403
+ "loss": 0.4214,
2404
+ "step": 341
2405
+ },
2406
+ {
2407
+ "epoch": 0.512743628185907,
2408
+ "grad_norm": 0.26799577624453275,
2409
+ "learning_rate": 9.491205990384915e-05,
2410
+ "loss": 0.3558,
2411
+ "step": 342
2412
+ },
2413
+ {
2414
+ "epoch": 0.5142428785607196,
2415
+ "grad_norm": 0.31884209355865095,
2416
+ "learning_rate": 9.487641497678723e-05,
2417
+ "loss": 0.283,
2418
+ "step": 343
2419
+ },
2420
+ {
2421
+ "epoch": 0.5157421289355323,
2422
+ "grad_norm": 0.25610524929335116,
2423
+ "learning_rate": 9.484065236655866e-05,
2424
+ "loss": 0.3559,
2425
+ "step": 344
2426
+ },
2427
+ {
2428
+ "epoch": 0.5172413793103449,
2429
+ "grad_norm": 0.27322281488042166,
2430
+ "learning_rate": 9.480477216694673e-05,
2431
+ "loss": 0.3612,
2432
+ "step": 345
2433
+ },
2434
+ {
2435
+ "epoch": 0.5187406296851574,
2436
+ "grad_norm": 0.3626964747600608,
2437
+ "learning_rate": 9.476877447204308e-05,
2438
+ "loss": 0.4505,
2439
+ "step": 346
2440
+ },
2441
+ {
2442
+ "epoch": 0.52023988005997,
2443
+ "grad_norm": 0.27115902974686,
2444
+ "learning_rate": 9.473265937624746e-05,
2445
+ "loss": 0.3331,
2446
+ "step": 347
2447
+ },
2448
+ {
2449
+ "epoch": 0.5217391304347826,
2450
+ "grad_norm": 0.2911124105978724,
2451
+ "learning_rate": 9.469642697426751e-05,
2452
+ "loss": 0.3335,
2453
+ "step": 348
2454
+ },
2455
+ {
2456
+ "epoch": 0.5232383808095952,
2457
+ "grad_norm": 0.2630412298342982,
2458
+ "learning_rate": 9.466007736111847e-05,
2459
+ "loss": 0.3523,
2460
+ "step": 349
2461
+ },
2462
+ {
2463
+ "epoch": 0.5247376311844077,
2464
+ "grad_norm": 0.3021916096793187,
2465
+ "learning_rate": 9.462361063212296e-05,
2466
+ "loss": 0.3385,
2467
+ "step": 350
2468
+ },
2469
+ {
2470
+ "epoch": 0.5262368815592204,
2471
+ "grad_norm": 0.26102211012329185,
2472
+ "learning_rate": 9.458702688291073e-05,
2473
+ "loss": 0.3887,
2474
+ "step": 351
2475
+ },
2476
+ {
2477
+ "epoch": 0.527736131934033,
2478
+ "grad_norm": 0.31331672829420604,
2479
+ "learning_rate": 9.45503262094184e-05,
2480
+ "loss": 0.3808,
2481
+ "step": 352
2482
+ },
2483
+ {
2484
+ "epoch": 0.5292353823088456,
2485
+ "grad_norm": 0.3395937475143066,
2486
+ "learning_rate": 9.45135087078892e-05,
2487
+ "loss": 0.3929,
2488
+ "step": 353
2489
+ },
2490
+ {
2491
+ "epoch": 0.5307346326836582,
2492
+ "grad_norm": 0.313000642060411,
2493
+ "learning_rate": 9.447657447487276e-05,
2494
+ "loss": 0.3232,
2495
+ "step": 354
2496
+ },
2497
+ {
2498
+ "epoch": 0.5322338830584707,
2499
+ "grad_norm": 0.2569177298354711,
2500
+ "learning_rate": 9.443952360722477e-05,
2501
+ "loss": 0.3375,
2502
+ "step": 355
2503
+ },
2504
+ {
2505
+ "epoch": 0.5337331334332833,
2506
+ "grad_norm": 0.298625266629892,
2507
+ "learning_rate": 9.440235620210683e-05,
2508
+ "loss": 0.3221,
2509
+ "step": 356
2510
+ },
2511
+ {
2512
+ "epoch": 0.5352323838080959,
2513
+ "grad_norm": 0.28339542679715557,
2514
+ "learning_rate": 9.436507235698612e-05,
2515
+ "loss": 0.3809,
2516
+ "step": 357
2517
+ },
2518
+ {
2519
+ "epoch": 0.5367316341829086,
2520
+ "grad_norm": 0.23099966195777122,
2521
+ "learning_rate": 9.432767216963517e-05,
2522
+ "loss": 0.3158,
2523
+ "step": 358
2524
+ },
2525
+ {
2526
+ "epoch": 0.5382308845577212,
2527
+ "grad_norm": 0.2033821064332453,
2528
+ "learning_rate": 9.429015573813163e-05,
2529
+ "loss": 0.3353,
2530
+ "step": 359
2531
+ },
2532
+ {
2533
+ "epoch": 0.5397301349325337,
2534
+ "grad_norm": 0.24785981271292937,
2535
+ "learning_rate": 9.425252316085796e-05,
2536
+ "loss": 0.2491,
2537
+ "step": 360
2538
+ },
2539
+ {
2540
+ "epoch": 0.5412293853073463,
2541
+ "grad_norm": 0.29083638154911723,
2542
+ "learning_rate": 9.421477453650118e-05,
2543
+ "loss": 0.4223,
2544
+ "step": 361
2545
+ },
2546
+ {
2547
+ "epoch": 0.5427286356821589,
2548
+ "grad_norm": 0.21257292314827547,
2549
+ "learning_rate": 9.41769099640527e-05,
2550
+ "loss": 0.2786,
2551
+ "step": 362
2552
+ },
2553
+ {
2554
+ "epoch": 0.5442278860569715,
2555
+ "grad_norm": 0.36509923092260005,
2556
+ "learning_rate": 9.413892954280792e-05,
2557
+ "loss": 0.3616,
2558
+ "step": 363
2559
+ },
2560
+ {
2561
+ "epoch": 0.545727136431784,
2562
+ "grad_norm": 0.2957566985927822,
2563
+ "learning_rate": 9.410083337236608e-05,
2564
+ "loss": 0.453,
2565
+ "step": 364
2566
+ },
2567
+ {
2568
+ "epoch": 0.5472263868065967,
2569
+ "grad_norm": 0.2691318840584187,
2570
+ "learning_rate": 9.406262155262994e-05,
2571
+ "loss": 0.3754,
2572
+ "step": 365
2573
+ },
2574
+ {
2575
+ "epoch": 0.5487256371814093,
2576
+ "grad_norm": 0.25149904950813456,
2577
+ "learning_rate": 9.402429418380554e-05,
2578
+ "loss": 0.323,
2579
+ "step": 366
2580
+ },
2581
+ {
2582
+ "epoch": 0.5502248875562219,
2583
+ "grad_norm": 0.2840588986769098,
2584
+ "learning_rate": 9.398585136640194e-05,
2585
+ "loss": 0.4011,
2586
+ "step": 367
2587
+ },
2588
+ {
2589
+ "epoch": 0.5517241379310345,
2590
+ "grad_norm": 0.2564982569670918,
2591
+ "learning_rate": 9.394729320123095e-05,
2592
+ "loss": 0.3975,
2593
+ "step": 368
2594
+ },
2595
+ {
2596
+ "epoch": 0.553223388305847,
2597
+ "grad_norm": 0.27983839424012363,
2598
+ "learning_rate": 9.390861978940686e-05,
2599
+ "loss": 0.2853,
2600
+ "step": 369
2601
+ },
2602
+ {
2603
+ "epoch": 0.5547226386806596,
2604
+ "grad_norm": 0.28846318704589047,
2605
+ "learning_rate": 9.386983123234618e-05,
2606
+ "loss": 0.3586,
2607
+ "step": 370
2608
+ },
2609
+ {
2610
+ "epoch": 0.5562218890554723,
2611
+ "grad_norm": 0.2878974141066758,
2612
+ "learning_rate": 9.38309276317674e-05,
2613
+ "loss": 0.3436,
2614
+ "step": 371
2615
+ },
2616
+ {
2617
+ "epoch": 0.5577211394302849,
2618
+ "grad_norm": 0.3165010840457768,
2619
+ "learning_rate": 9.379190908969064e-05,
2620
+ "loss": 0.4134,
2621
+ "step": 372
2622
+ },
2623
+ {
2624
+ "epoch": 0.5592203898050975,
2625
+ "grad_norm": 0.29437989979526274,
2626
+ "learning_rate": 9.37527757084375e-05,
2627
+ "loss": 0.4321,
2628
+ "step": 373
2629
+ },
2630
+ {
2631
+ "epoch": 0.56071964017991,
2632
+ "grad_norm": 0.23707887450134235,
2633
+ "learning_rate": 9.37135275906307e-05,
2634
+ "loss": 0.2919,
2635
+ "step": 374
2636
+ },
2637
+ {
2638
+ "epoch": 0.5622188905547226,
2639
+ "grad_norm": 0.26492722235239113,
2640
+ "learning_rate": 9.367416483919387e-05,
2641
+ "loss": 0.3595,
2642
+ "step": 375
2643
+ },
2644
+ {
2645
+ "epoch": 0.5637181409295352,
2646
+ "grad_norm": 0.3278394329102156,
2647
+ "learning_rate": 9.363468755735123e-05,
2648
+ "loss": 0.4638,
2649
+ "step": 376
2650
+ },
2651
+ {
2652
+ "epoch": 0.5652173913043478,
2653
+ "grad_norm": 0.274302845936504,
2654
+ "learning_rate": 9.359509584862736e-05,
2655
+ "loss": 0.3757,
2656
+ "step": 377
2657
+ },
2658
+ {
2659
+ "epoch": 0.5667166416791605,
2660
+ "grad_norm": 0.24825772185244388,
2661
+ "learning_rate": 9.355538981684687e-05,
2662
+ "loss": 0.3122,
2663
+ "step": 378
2664
+ },
2665
+ {
2666
+ "epoch": 0.568215892053973,
2667
+ "grad_norm": 0.28724839077128644,
2668
+ "learning_rate": 9.351556956613423e-05,
2669
+ "loss": 0.3233,
2670
+ "step": 379
2671
+ },
2672
+ {
2673
+ "epoch": 0.5697151424287856,
2674
+ "grad_norm": 0.24621764327015722,
2675
+ "learning_rate": 9.347563520091337e-05,
2676
+ "loss": 0.3305,
2677
+ "step": 380
2678
+ },
2679
+ {
2680
+ "epoch": 0.5712143928035982,
2681
+ "grad_norm": 0.2522265573154381,
2682
+ "learning_rate": 9.343558682590756e-05,
2683
+ "loss": 0.3637,
2684
+ "step": 381
2685
+ },
2686
+ {
2687
+ "epoch": 0.5727136431784108,
2688
+ "grad_norm": 0.32631779983370734,
2689
+ "learning_rate": 9.339542454613895e-05,
2690
+ "loss": 0.3284,
2691
+ "step": 382
2692
+ },
2693
+ {
2694
+ "epoch": 0.5742128935532234,
2695
+ "grad_norm": 0.2346578706532781,
2696
+ "learning_rate": 9.335514846692845e-05,
2697
+ "loss": 0.3684,
2698
+ "step": 383
2699
+ },
2700
+ {
2701
+ "epoch": 0.5757121439280359,
2702
+ "grad_norm": 0.246495791651111,
2703
+ "learning_rate": 9.331475869389538e-05,
2704
+ "loss": 0.3512,
2705
+ "step": 384
2706
+ },
2707
+ {
2708
+ "epoch": 0.5772113943028486,
2709
+ "grad_norm": 0.2553654567157137,
2710
+ "learning_rate": 9.327425533295724e-05,
2711
+ "loss": 0.3332,
2712
+ "step": 385
2713
+ },
2714
+ {
2715
+ "epoch": 0.5787106446776612,
2716
+ "grad_norm": 0.24218388437087565,
2717
+ "learning_rate": 9.323363849032933e-05,
2718
+ "loss": 0.3537,
2719
+ "step": 386
2720
+ },
2721
+ {
2722
+ "epoch": 0.5802098950524738,
2723
+ "grad_norm": 0.24206819796148848,
2724
+ "learning_rate": 9.31929082725246e-05,
2725
+ "loss": 0.3135,
2726
+ "step": 387
2727
+ },
2728
+ {
2729
+ "epoch": 0.5817091454272864,
2730
+ "grad_norm": 0.21920989707665034,
2731
+ "learning_rate": 9.31520647863533e-05,
2732
+ "loss": 0.2573,
2733
+ "step": 388
2734
+ },
2735
+ {
2736
+ "epoch": 0.5832083958020989,
2737
+ "grad_norm": 0.26205762124974163,
2738
+ "learning_rate": 9.31111081389227e-05,
2739
+ "loss": 0.3242,
2740
+ "step": 389
2741
+ },
2742
+ {
2743
+ "epoch": 0.5847076461769115,
2744
+ "grad_norm": 0.2933602247572226,
2745
+ "learning_rate": 9.307003843763686e-05,
2746
+ "loss": 0.3434,
2747
+ "step": 390
2748
+ },
2749
+ {
2750
+ "epoch": 0.5862068965517241,
2751
+ "grad_norm": 0.3440282968418603,
2752
+ "learning_rate": 9.302885579019627e-05,
2753
+ "loss": 0.3393,
2754
+ "step": 391
2755
+ },
2756
+ {
2757
+ "epoch": 0.5877061469265368,
2758
+ "grad_norm": 0.24817486584899429,
2759
+ "learning_rate": 9.298756030459761e-05,
2760
+ "loss": 0.3126,
2761
+ "step": 392
2762
+ },
2763
+ {
2764
+ "epoch": 0.5892053973013494,
2765
+ "grad_norm": 0.2927645338355415,
2766
+ "learning_rate": 9.294615208913348e-05,
2767
+ "loss": 0.3573,
2768
+ "step": 393
2769
+ },
2770
+ {
2771
+ "epoch": 0.5907046476761619,
2772
+ "grad_norm": 0.2849254547785852,
2773
+ "learning_rate": 9.290463125239213e-05,
2774
+ "loss": 0.2959,
2775
+ "step": 394
2776
+ },
2777
+ {
2778
+ "epoch": 0.5922038980509745,
2779
+ "grad_norm": 0.2188112015794616,
2780
+ "learning_rate": 9.286299790325708e-05,
2781
+ "loss": 0.3355,
2782
+ "step": 395
2783
+ },
2784
+ {
2785
+ "epoch": 0.5937031484257871,
2786
+ "grad_norm": 0.3050559512694953,
2787
+ "learning_rate": 9.282125215090694e-05,
2788
+ "loss": 0.3129,
2789
+ "step": 396
2790
+ },
2791
+ {
2792
+ "epoch": 0.5952023988005997,
2793
+ "grad_norm": 0.272407011380095,
2794
+ "learning_rate": 9.277939410481507e-05,
2795
+ "loss": 0.302,
2796
+ "step": 397
2797
+ },
2798
+ {
2799
+ "epoch": 0.5967016491754122,
2800
+ "grad_norm": 0.3080176206091004,
2801
+ "learning_rate": 9.273742387474934e-05,
2802
+ "loss": 0.3836,
2803
+ "step": 398
2804
+ },
2805
+ {
2806
+ "epoch": 0.5982008995502249,
2807
+ "grad_norm": 0.23918167205305313,
2808
+ "learning_rate": 9.269534157077177e-05,
2809
+ "loss": 0.2855,
2810
+ "step": 399
2811
+ },
2812
+ {
2813
+ "epoch": 0.5997001499250375,
2814
+ "grad_norm": 0.24438525396104838,
2815
+ "learning_rate": 9.265314730323829e-05,
2816
+ "loss": 0.3112,
2817
+ "step": 400
2818
+ },
2819
+ {
2820
+ "epoch": 0.5997001499250375,
2821
+ "eval_loss": 0.3407685160636902,
2822
+ "eval_runtime": 9.3927,
2823
+ "eval_samples_per_second": 5.749,
2824
+ "eval_steps_per_second": 1.491,
2825
+ "step": 400
2826
+ },
2827
+ {
2828
+ "epoch": 0.6011994002998501,
2829
+ "grad_norm": 0.28086948255861754,
2830
+ "learning_rate": 9.261084118279847e-05,
2831
+ "loss": 0.3563,
2832
+ "step": 401
2833
+ },
2834
+ {
2835
+ "epoch": 0.6026986506746627,
2836
+ "grad_norm": 0.22935815273555676,
2837
+ "learning_rate": 9.256842332039518e-05,
2838
+ "loss": 0.3121,
2839
+ "step": 402
2840
+ },
2841
+ {
2842
+ "epoch": 0.6041979010494752,
2843
+ "grad_norm": 0.260216111143008,
2844
+ "learning_rate": 9.252589382726426e-05,
2845
+ "loss": 0.313,
2846
+ "step": 403
2847
+ },
2848
+ {
2849
+ "epoch": 0.6056971514242878,
2850
+ "grad_norm": 0.2935638547580935,
2851
+ "learning_rate": 9.248325281493444e-05,
2852
+ "loss": 0.2804,
2853
+ "step": 404
2854
+ },
2855
+ {
2856
+ "epoch": 0.6071964017991005,
2857
+ "grad_norm": 0.27940879478367364,
2858
+ "learning_rate": 9.244050039522672e-05,
2859
+ "loss": 0.3349,
2860
+ "step": 405
2861
+ },
2862
+ {
2863
+ "epoch": 0.6086956521739131,
2864
+ "grad_norm": 0.29625175851309354,
2865
+ "learning_rate": 9.239763668025439e-05,
2866
+ "loss": 0.3378,
2867
+ "step": 406
2868
+ },
2869
+ {
2870
+ "epoch": 0.6101949025487257,
2871
+ "grad_norm": 0.31688242986748677,
2872
+ "learning_rate": 9.235466178242254e-05,
2873
+ "loss": 0.3666,
2874
+ "step": 407
2875
+ },
2876
+ {
2877
+ "epoch": 0.6116941529235382,
2878
+ "grad_norm": 0.31656653083486364,
2879
+ "learning_rate": 9.231157581442781e-05,
2880
+ "loss": 0.3644,
2881
+ "step": 408
2882
+ },
2883
+ {
2884
+ "epoch": 0.6131934032983508,
2885
+ "grad_norm": 0.24143659030082,
2886
+ "learning_rate": 9.226837888925813e-05,
2887
+ "loss": 0.3533,
2888
+ "step": 409
2889
+ },
2890
+ {
2891
+ "epoch": 0.6146926536731634,
2892
+ "grad_norm": 0.26124787091307305,
2893
+ "learning_rate": 9.222507112019242e-05,
2894
+ "loss": 0.3271,
2895
+ "step": 410
2896
+ },
2897
+ {
2898
+ "epoch": 0.616191904047976,
2899
+ "grad_norm": 0.31324865522778583,
2900
+ "learning_rate": 9.218165262080023e-05,
2901
+ "loss": 0.3428,
2902
+ "step": 411
2903
+ },
2904
+ {
2905
+ "epoch": 0.6176911544227887,
2906
+ "grad_norm": 0.3652798207487008,
2907
+ "learning_rate": 9.213812350494153e-05,
2908
+ "loss": 0.3388,
2909
+ "step": 412
2910
+ },
2911
+ {
2912
+ "epoch": 0.6191904047976012,
2913
+ "grad_norm": 0.28885436126771186,
2914
+ "learning_rate": 9.209448388676635e-05,
2915
+ "loss": 0.3549,
2916
+ "step": 413
2917
+ },
2918
+ {
2919
+ "epoch": 0.6206896551724138,
2920
+ "grad_norm": 0.24086819662287953,
2921
+ "learning_rate": 9.20507338807145e-05,
2922
+ "loss": 0.339,
2923
+ "step": 414
2924
+ },
2925
+ {
2926
+ "epoch": 0.6221889055472264,
2927
+ "grad_norm": 0.3069971378954431,
2928
+ "learning_rate": 9.200687360151528e-05,
2929
+ "loss": 0.2951,
2930
+ "step": 415
2931
+ },
2932
+ {
2933
+ "epoch": 0.623688155922039,
2934
+ "grad_norm": 0.2746888981608114,
2935
+ "learning_rate": 9.196290316418711e-05,
2936
+ "loss": 0.3258,
2937
+ "step": 416
2938
+ },
2939
+ {
2940
+ "epoch": 0.6251874062968515,
2941
+ "grad_norm": 0.2529325040400764,
2942
+ "learning_rate": 9.191882268403743e-05,
2943
+ "loss": 0.2876,
2944
+ "step": 417
2945
+ },
2946
+ {
2947
+ "epoch": 0.6266866566716641,
2948
+ "grad_norm": 0.2862769095319526,
2949
+ "learning_rate": 9.187463227666205e-05,
2950
+ "loss": 0.2936,
2951
+ "step": 418
2952
+ },
2953
+ {
2954
+ "epoch": 0.6281859070464768,
2955
+ "grad_norm": 0.2817150674471743,
2956
+ "learning_rate": 9.183033205794525e-05,
2957
+ "loss": 0.3294,
2958
+ "step": 419
2959
+ },
2960
+ {
2961
+ "epoch": 0.6296851574212894,
2962
+ "grad_norm": 0.2893136777786241,
2963
+ "learning_rate": 9.178592214405914e-05,
2964
+ "loss": 0.3312,
2965
+ "step": 420
2966
+ },
2967
+ {
2968
+ "epoch": 0.631184407796102,
2969
+ "grad_norm": 0.26929771941251335,
2970
+ "learning_rate": 9.174140265146356e-05,
2971
+ "loss": 0.3008,
2972
+ "step": 421
2973
+ },
2974
+ {
2975
+ "epoch": 0.6326836581709145,
2976
+ "grad_norm": 0.2868917422770835,
2977
+ "learning_rate": 9.16967736969057e-05,
2978
+ "loss": 0.3885,
2979
+ "step": 422
2980
+ },
2981
+ {
2982
+ "epoch": 0.6341829085457271,
2983
+ "grad_norm": 0.2885809245551549,
2984
+ "learning_rate": 9.165203539741976e-05,
2985
+ "loss": 0.2753,
2986
+ "step": 423
2987
+ },
2988
+ {
2989
+ "epoch": 0.6356821589205397,
2990
+ "grad_norm": 0.31004674562262013,
2991
+ "learning_rate": 9.160718787032674e-05,
2992
+ "loss": 0.3216,
2993
+ "step": 424
2994
+ },
2995
+ {
2996
+ "epoch": 0.6371814092953523,
2997
+ "grad_norm": 0.27639070167163965,
2998
+ "learning_rate": 9.156223123323405e-05,
2999
+ "loss": 0.337,
3000
+ "step": 425
3001
+ },
3002
+ {
3003
+ "epoch": 0.638680659670165,
3004
+ "grad_norm": 0.24277290762336287,
3005
+ "learning_rate": 9.15171656040352e-05,
3006
+ "loss": 0.2717,
3007
+ "step": 426
3008
+ },
3009
+ {
3010
+ "epoch": 0.6401799100449775,
3011
+ "grad_norm": 0.28040799206930017,
3012
+ "learning_rate": 9.147199110090959e-05,
3013
+ "loss": 0.2856,
3014
+ "step": 427
3015
+ },
3016
+ {
3017
+ "epoch": 0.6416791604197901,
3018
+ "grad_norm": 0.24438190494058687,
3019
+ "learning_rate": 9.142670784232207e-05,
3020
+ "loss": 0.3202,
3021
+ "step": 428
3022
+ },
3023
+ {
3024
+ "epoch": 0.6431784107946027,
3025
+ "grad_norm": 0.23933976786717923,
3026
+ "learning_rate": 9.13813159470227e-05,
3027
+ "loss": 0.2508,
3028
+ "step": 429
3029
+ },
3030
+ {
3031
+ "epoch": 0.6446776611694153,
3032
+ "grad_norm": 0.28109080164156414,
3033
+ "learning_rate": 9.133581553404644e-05,
3034
+ "loss": 0.3305,
3035
+ "step": 430
3036
+ },
3037
+ {
3038
+ "epoch": 0.6461769115442278,
3039
+ "grad_norm": 0.30768631307262345,
3040
+ "learning_rate": 9.129020672271283e-05,
3041
+ "loss": 0.3597,
3042
+ "step": 431
3043
+ },
3044
+ {
3045
+ "epoch": 0.6476761619190404,
3046
+ "grad_norm": 0.26591356505593744,
3047
+ "learning_rate": 9.124448963262565e-05,
3048
+ "loss": 0.3408,
3049
+ "step": 432
3050
+ },
3051
+ {
3052
+ "epoch": 0.6491754122938531,
3053
+ "grad_norm": 0.32694872042068657,
3054
+ "learning_rate": 9.119866438367263e-05,
3055
+ "loss": 0.3691,
3056
+ "step": 433
3057
+ },
3058
+ {
3059
+ "epoch": 0.6506746626686657,
3060
+ "grad_norm": 0.26660416233614764,
3061
+ "learning_rate": 9.115273109602516e-05,
3062
+ "loss": 0.3439,
3063
+ "step": 434
3064
+ },
3065
+ {
3066
+ "epoch": 0.6521739130434783,
3067
+ "grad_norm": 0.26328826374189573,
3068
+ "learning_rate": 9.110668989013791e-05,
3069
+ "loss": 0.3045,
3070
+ "step": 435
3071
+ },
3072
+ {
3073
+ "epoch": 0.6536731634182908,
3074
+ "grad_norm": 0.29335925103622074,
3075
+ "learning_rate": 9.10605408867486e-05,
3076
+ "loss": 0.3191,
3077
+ "step": 436
3078
+ },
3079
+ {
3080
+ "epoch": 0.6551724137931034,
3081
+ "grad_norm": 0.31652933052287996,
3082
+ "learning_rate": 9.101428420687759e-05,
3083
+ "loss": 0.3203,
3084
+ "step": 437
3085
+ },
3086
+ {
3087
+ "epoch": 0.656671664167916,
3088
+ "grad_norm": 0.2326421080227109,
3089
+ "learning_rate": 9.096791997182765e-05,
3090
+ "loss": 0.3212,
3091
+ "step": 438
3092
+ },
3093
+ {
3094
+ "epoch": 0.6581709145427287,
3095
+ "grad_norm": 0.30373914992832884,
3096
+ "learning_rate": 9.092144830318358e-05,
3097
+ "loss": 0.3929,
3098
+ "step": 439
3099
+ },
3100
+ {
3101
+ "epoch": 0.6596701649175413,
3102
+ "grad_norm": 0.227479352264213,
3103
+ "learning_rate": 9.08748693228119e-05,
3104
+ "loss": 0.2975,
3105
+ "step": 440
3106
+ },
3107
+ {
3108
+ "epoch": 0.6611694152923538,
3109
+ "grad_norm": 0.32345871664484027,
3110
+ "learning_rate": 9.082818315286055e-05,
3111
+ "loss": 0.3546,
3112
+ "step": 441
3113
+ },
3114
+ {
3115
+ "epoch": 0.6626686656671664,
3116
+ "grad_norm": 0.2834693925576374,
3117
+ "learning_rate": 9.07813899157586e-05,
3118
+ "loss": 0.353,
3119
+ "step": 442
3120
+ },
3121
+ {
3122
+ "epoch": 0.664167916041979,
3123
+ "grad_norm": 0.2500144651231018,
3124
+ "learning_rate": 9.073448973421582e-05,
3125
+ "loss": 0.2895,
3126
+ "step": 443
3127
+ },
3128
+ {
3129
+ "epoch": 0.6656671664167916,
3130
+ "grad_norm": 0.28489985439629356,
3131
+ "learning_rate": 9.068748273122249e-05,
3132
+ "loss": 0.3277,
3133
+ "step": 444
3134
+ },
3135
+ {
3136
+ "epoch": 0.6671664167916042,
3137
+ "grad_norm": 0.30259215341738166,
3138
+ "learning_rate": 9.0640369030049e-05,
3139
+ "loss": 0.3431,
3140
+ "step": 445
3141
+ },
3142
+ {
3143
+ "epoch": 0.6686656671664168,
3144
+ "grad_norm": 0.32367511355388184,
3145
+ "learning_rate": 9.059314875424553e-05,
3146
+ "loss": 0.3839,
3147
+ "step": 446
3148
+ },
3149
+ {
3150
+ "epoch": 0.6701649175412294,
3151
+ "grad_norm": 0.30859477200629365,
3152
+ "learning_rate": 9.054582202764175e-05,
3153
+ "loss": 0.4615,
3154
+ "step": 447
3155
+ },
3156
+ {
3157
+ "epoch": 0.671664167916042,
3158
+ "grad_norm": 0.29938499377624256,
3159
+ "learning_rate": 9.049838897434648e-05,
3160
+ "loss": 0.3154,
3161
+ "step": 448
3162
+ },
3163
+ {
3164
+ "epoch": 0.6731634182908546,
3165
+ "grad_norm": 0.27664303623411163,
3166
+ "learning_rate": 9.045084971874738e-05,
3167
+ "loss": 0.3022,
3168
+ "step": 449
3169
+ },
3170
+ {
3171
+ "epoch": 0.6746626686656672,
3172
+ "grad_norm": 0.22798315434848834,
3173
+ "learning_rate": 9.04032043855106e-05,
3174
+ "loss": 0.3253,
3175
+ "step": 450
3176
+ },
3177
+ {
3178
+ "epoch": 0.6761619190404797,
3179
+ "grad_norm": 0.23774577554676588,
3180
+ "learning_rate": 9.035545309958046e-05,
3181
+ "loss": 0.2812,
3182
+ "step": 451
3183
+ },
3184
+ {
3185
+ "epoch": 0.6776611694152923,
3186
+ "grad_norm": 0.2506771654678938,
3187
+ "learning_rate": 9.030759598617918e-05,
3188
+ "loss": 0.3571,
3189
+ "step": 452
3190
+ },
3191
+ {
3192
+ "epoch": 0.679160419790105,
3193
+ "grad_norm": 0.25762084720123335,
3194
+ "learning_rate": 9.025963317080641e-05,
3195
+ "loss": 0.3227,
3196
+ "step": 453
3197
+ },
3198
+ {
3199
+ "epoch": 0.6806596701649176,
3200
+ "grad_norm": 0.3014366864803227,
3201
+ "learning_rate": 9.021156477923909e-05,
3202
+ "loss": 0.3736,
3203
+ "step": 454
3204
+ },
3205
+ {
3206
+ "epoch": 0.6821589205397302,
3207
+ "grad_norm": 0.2228445980502147,
3208
+ "learning_rate": 9.016339093753093e-05,
3209
+ "loss": 0.3036,
3210
+ "step": 455
3211
+ },
3212
+ {
3213
+ "epoch": 0.6836581709145427,
3214
+ "grad_norm": 0.3401829022565116,
3215
+ "learning_rate": 9.011511177201225e-05,
3216
+ "loss": 0.3454,
3217
+ "step": 456
3218
+ },
3219
+ {
3220
+ "epoch": 0.6851574212893553,
3221
+ "grad_norm": 0.22934951098956732,
3222
+ "learning_rate": 9.006672740928952e-05,
3223
+ "loss": 0.2689,
3224
+ "step": 457
3225
+ },
3226
+ {
3227
+ "epoch": 0.6866566716641679,
3228
+ "grad_norm": 0.22478386258913607,
3229
+ "learning_rate": 9.001823797624506e-05,
3230
+ "loss": 0.3295,
3231
+ "step": 458
3232
+ },
3233
+ {
3234
+ "epoch": 0.6881559220389805,
3235
+ "grad_norm": 0.3526430665907978,
3236
+ "learning_rate": 8.99696436000368e-05,
3237
+ "loss": 0.3868,
3238
+ "step": 459
3239
+ },
3240
+ {
3241
+ "epoch": 0.6896551724137931,
3242
+ "grad_norm": 0.28621239134868454,
3243
+ "learning_rate": 8.992094440809782e-05,
3244
+ "loss": 0.3581,
3245
+ "step": 460
3246
+ },
3247
+ {
3248
+ "epoch": 0.6911544227886057,
3249
+ "grad_norm": 0.3363379390019984,
3250
+ "learning_rate": 8.987214052813604e-05,
3251
+ "loss": 0.4145,
3252
+ "step": 461
3253
+ },
3254
+ {
3255
+ "epoch": 0.6926536731634183,
3256
+ "grad_norm": 0.2351530418979157,
3257
+ "learning_rate": 8.982323208813399e-05,
3258
+ "loss": 0.2773,
3259
+ "step": 462
3260
+ },
3261
+ {
3262
+ "epoch": 0.6941529235382309,
3263
+ "grad_norm": 0.29565189621212185,
3264
+ "learning_rate": 8.977421921634832e-05,
3265
+ "loss": 0.4183,
3266
+ "step": 463
3267
+ },
3268
+ {
3269
+ "epoch": 0.6956521739130435,
3270
+ "grad_norm": 0.2788029853802518,
3271
+ "learning_rate": 8.972510204130958e-05,
3272
+ "loss": 0.2822,
3273
+ "step": 464
3274
+ },
3275
+ {
3276
+ "epoch": 0.697151424287856,
3277
+ "grad_norm": 0.3235950747830869,
3278
+ "learning_rate": 8.967588069182185e-05,
3279
+ "loss": 0.3815,
3280
+ "step": 465
3281
+ },
3282
+ {
3283
+ "epoch": 0.6986506746626686,
3284
+ "grad_norm": 0.28959339322420985,
3285
+ "learning_rate": 8.962655529696236e-05,
3286
+ "loss": 0.3335,
3287
+ "step": 466
3288
+ },
3289
+ {
3290
+ "epoch": 0.7001499250374813,
3291
+ "grad_norm": 0.3484195937705581,
3292
+ "learning_rate": 8.957712598608123e-05,
3293
+ "loss": 0.3882,
3294
+ "step": 467
3295
+ },
3296
+ {
3297
+ "epoch": 0.7016491754122939,
3298
+ "grad_norm": 0.22194351726641875,
3299
+ "learning_rate": 8.952759288880104e-05,
3300
+ "loss": 0.253,
3301
+ "step": 468
3302
+ },
3303
+ {
3304
+ "epoch": 0.7031484257871065,
3305
+ "grad_norm": 0.227874764604873,
3306
+ "learning_rate": 8.947795613501658e-05,
3307
+ "loss": 0.2461,
3308
+ "step": 469
3309
+ },
3310
+ {
3311
+ "epoch": 0.704647676161919,
3312
+ "grad_norm": 0.27048930273058597,
3313
+ "learning_rate": 8.942821585489445e-05,
3314
+ "loss": 0.343,
3315
+ "step": 470
3316
+ },
3317
+ {
3318
+ "epoch": 0.7061469265367316,
3319
+ "grad_norm": 0.3359789403389439,
3320
+ "learning_rate": 8.937837217887273e-05,
3321
+ "loss": 0.319,
3322
+ "step": 471
3323
+ },
3324
+ {
3325
+ "epoch": 0.7076461769115442,
3326
+ "grad_norm": 0.37706582391715526,
3327
+ "learning_rate": 8.932842523766065e-05,
3328
+ "loss": 0.3217,
3329
+ "step": 472
3330
+ },
3331
+ {
3332
+ "epoch": 0.7091454272863568,
3333
+ "grad_norm": 0.2782992746944311,
3334
+ "learning_rate": 8.927837516223824e-05,
3335
+ "loss": 0.3403,
3336
+ "step": 473
3337
+ },
3338
+ {
3339
+ "epoch": 0.7106446776611695,
3340
+ "grad_norm": 0.28404551462352956,
3341
+ "learning_rate": 8.922822208385599e-05,
3342
+ "loss": 0.315,
3343
+ "step": 474
3344
+ },
3345
+ {
3346
+ "epoch": 0.712143928035982,
3347
+ "grad_norm": 0.2860491828984938,
3348
+ "learning_rate": 8.917796613403451e-05,
3349
+ "loss": 0.3458,
3350
+ "step": 475
3351
+ },
3352
+ {
3353
+ "epoch": 0.7136431784107946,
3354
+ "grad_norm": 0.3662312678834754,
3355
+ "learning_rate": 8.912760744456415e-05,
3356
+ "loss": 0.3931,
3357
+ "step": 476
3358
+ },
3359
+ {
3360
+ "epoch": 0.7151424287856072,
3361
+ "grad_norm": 0.31234791343768453,
3362
+ "learning_rate": 8.907714614750473e-05,
3363
+ "loss": 0.3634,
3364
+ "step": 477
3365
+ },
3366
+ {
3367
+ "epoch": 0.7166416791604198,
3368
+ "grad_norm": 0.29815368036130224,
3369
+ "learning_rate": 8.902658237518509e-05,
3370
+ "loss": 0.2963,
3371
+ "step": 478
3372
+ },
3373
+ {
3374
+ "epoch": 0.7181409295352323,
3375
+ "grad_norm": 0.23552021666504025,
3376
+ "learning_rate": 8.897591626020284e-05,
3377
+ "loss": 0.3405,
3378
+ "step": 479
3379
+ },
3380
+ {
3381
+ "epoch": 0.719640179910045,
3382
+ "grad_norm": 0.26121806297471106,
3383
+ "learning_rate": 8.892514793542397e-05,
3384
+ "loss": 0.3361,
3385
+ "step": 480
3386
+ },
3387
+ {
3388
+ "epoch": 0.7211394302848576,
3389
+ "grad_norm": 0.2236705711698251,
3390
+ "learning_rate": 8.887427753398248e-05,
3391
+ "loss": 0.2836,
3392
+ "step": 481
3393
+ },
3394
+ {
3395
+ "epoch": 0.7226386806596702,
3396
+ "grad_norm": 0.29173625954304344,
3397
+ "learning_rate": 8.882330518928006e-05,
3398
+ "loss": 0.2999,
3399
+ "step": 482
3400
+ },
3401
+ {
3402
+ "epoch": 0.7241379310344828,
3403
+ "grad_norm": 0.28149246277263024,
3404
+ "learning_rate": 8.877223103498575e-05,
3405
+ "loss": 0.3166,
3406
+ "step": 483
3407
+ },
3408
+ {
3409
+ "epoch": 0.7256371814092953,
3410
+ "grad_norm": 0.32348440311697224,
3411
+ "learning_rate": 8.872105520503559e-05,
3412
+ "loss": 0.2947,
3413
+ "step": 484
3414
+ },
3415
+ {
3416
+ "epoch": 0.7271364317841079,
3417
+ "grad_norm": 0.28498136767124405,
3418
+ "learning_rate": 8.86697778336322e-05,
3419
+ "loss": 0.3612,
3420
+ "step": 485
3421
+ },
3422
+ {
3423
+ "epoch": 0.7286356821589205,
3424
+ "grad_norm": 0.2065250953142658,
3425
+ "learning_rate": 8.861839905524452e-05,
3426
+ "loss": 0.2752,
3427
+ "step": 486
3428
+ },
3429
+ {
3430
+ "epoch": 0.7301349325337332,
3431
+ "grad_norm": 0.27922971095665367,
3432
+ "learning_rate": 8.856691900460739e-05,
3433
+ "loss": 0.3446,
3434
+ "step": 487
3435
+ },
3436
+ {
3437
+ "epoch": 0.7316341829085458,
3438
+ "grad_norm": 0.24554129661586624,
3439
+ "learning_rate": 8.851533781672125e-05,
3440
+ "loss": 0.3301,
3441
+ "step": 488
3442
+ },
3443
+ {
3444
+ "epoch": 0.7331334332833583,
3445
+ "grad_norm": 0.2787894552165883,
3446
+ "learning_rate": 8.846365562685177e-05,
3447
+ "loss": 0.3224,
3448
+ "step": 489
3449
+ },
3450
+ {
3451
+ "epoch": 0.7346326836581709,
3452
+ "grad_norm": 0.2609916330094101,
3453
+ "learning_rate": 8.841187257052944e-05,
3454
+ "loss": 0.3828,
3455
+ "step": 490
3456
+ },
3457
+ {
3458
+ "epoch": 0.7361319340329835,
3459
+ "grad_norm": 0.24196293824238985,
3460
+ "learning_rate": 8.835998878354931e-05,
3461
+ "loss": 0.3443,
3462
+ "step": 491
3463
+ },
3464
+ {
3465
+ "epoch": 0.7376311844077961,
3466
+ "grad_norm": 0.25681730610356324,
3467
+ "learning_rate": 8.830800440197055e-05,
3468
+ "loss": 0.298,
3469
+ "step": 492
3470
+ },
3471
+ {
3472
+ "epoch": 0.7391304347826086,
3473
+ "grad_norm": 0.25278746211884884,
3474
+ "learning_rate": 8.825591956211614e-05,
3475
+ "loss": 0.3513,
3476
+ "step": 493
3477
+ },
3478
+ {
3479
+ "epoch": 0.7406296851574213,
3480
+ "grad_norm": 0.29905568059270327,
3481
+ "learning_rate": 8.820373440057252e-05,
3482
+ "loss": 0.3411,
3483
+ "step": 494
3484
+ },
3485
+ {
3486
+ "epoch": 0.7421289355322339,
3487
+ "grad_norm": 0.25479992559156345,
3488
+ "learning_rate": 8.815144905418917e-05,
3489
+ "loss": 0.3006,
3490
+ "step": 495
3491
+ },
3492
+ {
3493
+ "epoch": 0.7436281859070465,
3494
+ "grad_norm": 0.30913694171782413,
3495
+ "learning_rate": 8.809906366007832e-05,
3496
+ "loss": 0.3306,
3497
+ "step": 496
3498
+ },
3499
+ {
3500
+ "epoch": 0.7451274362818591,
3501
+ "grad_norm": 0.3457652288832713,
3502
+ "learning_rate": 8.804657835561456e-05,
3503
+ "loss": 0.3289,
3504
+ "step": 497
3505
+ },
3506
+ {
3507
+ "epoch": 0.7466266866566716,
3508
+ "grad_norm": 0.27983670458975085,
3509
+ "learning_rate": 8.79939932784345e-05,
3510
+ "loss": 0.3474,
3511
+ "step": 498
3512
+ },
3513
+ {
3514
+ "epoch": 0.7481259370314842,
3515
+ "grad_norm": 0.26020086725871355,
3516
+ "learning_rate": 8.794130856643634e-05,
3517
+ "loss": 0.3125,
3518
+ "step": 499
3519
+ },
3520
+ {
3521
+ "epoch": 0.7496251874062968,
3522
+ "grad_norm": 0.25921150400062415,
3523
+ "learning_rate": 8.78885243577796e-05,
3524
+ "loss": 0.3206,
3525
+ "step": 500
3526
+ }
3527
+ ],
3528
+ "logging_steps": 1,
3529
+ "max_steps": 2001,
3530
+ "num_input_tokens_seen": 0,
3531
+ "num_train_epochs": 3,
3532
+ "save_steps": 250,
3533
+ "stateful_callbacks": {
3534
+ "TrainerControl": {
3535
+ "args": {
3536
+ "should_epoch_stop": false,
3537
+ "should_evaluate": false,
3538
+ "should_log": false,
3539
+ "should_save": true,
3540
+ "should_training_stop": false
3541
+ },
3542
+ "attributes": {}
3543
+ }
3544
+ },
3545
+ "total_flos": 346097400086528.0,
3546
+ "train_batch_size": 2,
3547
+ "trial_name": null,
3548
+ "trial_params": null
3549
+ }
training_args.bin ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:1810b92b16090910c0a215152d65748b8f3be6da52b74c787680b1b930f8fc68
3
+ size 7224
vocab.json ADDED
The diff for this file is too large to render. See raw diff
 
zero_to_fp32.py ADDED
@@ -0,0 +1,760 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ #!/usr/bin/env python
2
+
3
+ # Copyright (c) Microsoft Corporation.
4
+ # SPDX-License-Identifier: Apache-2.0
5
+
6
+ # DeepSpeed Team
7
+
8
+ # This script extracts fp32 consolidated weights from a zero 1, 2 and 3 DeepSpeed checkpoints. It gets
9
+ # copied into the top level checkpoint dir, so the user can easily do the conversion at any point in
10
+ # the future. Once extracted, the weights don't require DeepSpeed and can be used in any
11
+ # application.
12
+ #
13
+ # example:
14
+ # python zero_to_fp32.py . output_dir/
15
+ # or
16
+ # python zero_to_fp32.py . output_dir/ --safe_serialization
17
+
18
+ import argparse
19
+ import torch
20
+ import glob
21
+ import math
22
+ import os
23
+ import re
24
+ import gc
25
+ import json
26
+ import numpy as np
27
+ from tqdm import tqdm
28
+ from collections import OrderedDict
29
+ from dataclasses import dataclass
30
+
31
+ # while this script doesn't use deepspeed to recover data, since the checkpoints are pickled with
32
+ # DeepSpeed data structures it has to be available in the current python environment.
33
+ from deepspeed.utils import logger
34
+ from deepspeed.checkpoint.constants import (DS_VERSION, OPTIMIZER_STATE_DICT, SINGLE_PARTITION_OF_FP32_GROUPS,
35
+ FP32_FLAT_GROUPS, ZERO_STAGE, PARTITION_COUNT, PARAM_SHAPES, BUFFER_NAMES,
36
+ FROZEN_PARAM_SHAPES, FROZEN_PARAM_FRAGMENTS)
37
+
38
+
39
+ @dataclass
40
+ class zero_model_state:
41
+ buffers: dict()
42
+ param_shapes: dict()
43
+ shared_params: list
44
+ ds_version: int
45
+ frozen_param_shapes: dict()
46
+ frozen_param_fragments: dict()
47
+
48
+
49
+ debug = 0
50
+
51
+ # load to cpu
52
+ device = torch.device('cpu')
53
+
54
+
55
+ def atoi(text):
56
+ return int(text) if text.isdigit() else text
57
+
58
+
59
+ def natural_keys(text):
60
+ '''
61
+ alist.sort(key=natural_keys) sorts in human order
62
+ http://nedbatchelder.com/blog/200712/human_sorting.html
63
+ (See Toothy's implementation in the comments)
64
+ '''
65
+ return [atoi(c) for c in re.split(r'(\d+)', text)]
66
+
67
+
68
+ def get_model_state_file(checkpoint_dir, zero_stage):
69
+ if not os.path.isdir(checkpoint_dir):
70
+ raise FileNotFoundError(f"Directory '{checkpoint_dir}' doesn't exist")
71
+
72
+ # there should be only one file
73
+ if zero_stage <= 2:
74
+ file = os.path.join(checkpoint_dir, "mp_rank_00_model_states.pt")
75
+ elif zero_stage == 3:
76
+ file = os.path.join(checkpoint_dir, "zero_pp_rank_0_mp_rank_00_model_states.pt")
77
+
78
+ if not os.path.exists(file):
79
+ raise FileNotFoundError(f"can't find model states file at '{file}'")
80
+
81
+ return file
82
+
83
+
84
+ def get_checkpoint_files(checkpoint_dir, glob_pattern):
85
+ # XXX: need to test that this simple glob rule works for multi-node setup too
86
+ ckpt_files = sorted(glob.glob(os.path.join(checkpoint_dir, glob_pattern)), key=natural_keys)
87
+
88
+ if len(ckpt_files) == 0:
89
+ raise FileNotFoundError(f"can't find {glob_pattern} files in directory '{checkpoint_dir}'")
90
+
91
+ return ckpt_files
92
+
93
+
94
+ def get_optim_files(checkpoint_dir):
95
+ return get_checkpoint_files(checkpoint_dir, "*_optim_states.pt")
96
+
97
+
98
+ def get_model_state_files(checkpoint_dir):
99
+ return get_checkpoint_files(checkpoint_dir, "*_model_states.pt")
100
+
101
+
102
+ def parse_model_states(files):
103
+ zero_model_states = []
104
+ for file in files:
105
+ state_dict = torch.load(file, map_location=device, weights_only=False)
106
+
107
+ if BUFFER_NAMES not in state_dict:
108
+ raise ValueError(f"{file} is not a model state checkpoint")
109
+ buffer_names = state_dict[BUFFER_NAMES]
110
+ if debug:
111
+ print("Found buffers:", buffer_names)
112
+
113
+ # recover just the buffers while restoring them to fp32 if they were saved in fp16
114
+ buffers = {k: v.float() for k, v in state_dict["module"].items() if k in buffer_names}
115
+ param_shapes = state_dict[PARAM_SHAPES]
116
+
117
+ # collect parameters that are included in param_shapes
118
+ param_names = []
119
+ for s in param_shapes:
120
+ for name in s.keys():
121
+ param_names.append(name)
122
+
123
+ # update with frozen parameters
124
+ frozen_param_shapes = state_dict.get(FROZEN_PARAM_SHAPES, None)
125
+ if frozen_param_shapes is not None:
126
+ if debug:
127
+ print(f"Found frozen_param_shapes: {frozen_param_shapes}")
128
+ param_names += list(frozen_param_shapes.keys())
129
+
130
+ # handle shared params
131
+ shared_params = [[k, v] for k, v in state_dict["shared_params"].items()]
132
+
133
+ ds_version = state_dict.get(DS_VERSION, None)
134
+
135
+ frozen_param_fragments = state_dict.get(FROZEN_PARAM_FRAGMENTS, None)
136
+
137
+ z_model_state = zero_model_state(buffers=buffers,
138
+ param_shapes=param_shapes,
139
+ shared_params=shared_params,
140
+ ds_version=ds_version,
141
+ frozen_param_shapes=frozen_param_shapes,
142
+ frozen_param_fragments=frozen_param_fragments)
143
+ zero_model_states.append(z_model_state)
144
+
145
+ return zero_model_states
146
+
147
+
148
+ def parse_optim_states(files, ds_checkpoint_dir):
149
+ total_files = len(files)
150
+ state_dicts = []
151
+ for f in tqdm(files, desc='Loading checkpoint shards'):
152
+ state_dict = torch.load(f, map_location=device, mmap=True, weights_only=False)
153
+ # immediately discard the potentially huge 2 optimizer states as we only care for fp32 master weights
154
+ # and also handle the case where it was already removed by another helper script
155
+ state_dict["optimizer_state_dict"].pop("optimizer_state_dict", None)
156
+ state_dicts.append(state_dict)
157
+
158
+ if not ZERO_STAGE in state_dicts[0][OPTIMIZER_STATE_DICT]:
159
+ raise ValueError(f"{files[0]} is not a zero checkpoint")
160
+ zero_stage = state_dicts[0][OPTIMIZER_STATE_DICT][ZERO_STAGE]
161
+ world_size = state_dicts[0][OPTIMIZER_STATE_DICT][PARTITION_COUNT]
162
+
163
+ # For ZeRO-2 each param group can have different partition_count as data parallelism for expert
164
+ # parameters can be different from data parallelism for non-expert parameters. So we can just
165
+ # use the max of the partition_count to get the dp world_size.
166
+
167
+ if type(world_size) is list:
168
+ world_size = max(world_size)
169
+
170
+ if world_size != total_files:
171
+ raise ValueError(
172
+ f"Expected {world_size} of '*_optim_states.pt' under '{ds_checkpoint_dir}' but found {total_files} files. "
173
+ "Possibly due to an overwrite of an old checkpoint, or a checkpoint didn't get saved by one or more processes."
174
+ )
175
+
176
+ # the groups are named differently in each stage
177
+ if zero_stage <= 2:
178
+ fp32_groups_key = SINGLE_PARTITION_OF_FP32_GROUPS
179
+ elif zero_stage == 3:
180
+ fp32_groups_key = FP32_FLAT_GROUPS
181
+ else:
182
+ raise ValueError(f"unknown zero stage {zero_stage}")
183
+
184
+ fp32_flat_groups = [state_dicts[i][OPTIMIZER_STATE_DICT][fp32_groups_key] for i in range(len(state_dicts))]
185
+ return zero_stage, world_size, fp32_flat_groups
186
+
187
+
188
+ def _get_fp32_state_dict_from_zero_checkpoint(ds_checkpoint_dir, exclude_frozen_parameters):
189
+ """
190
+ Returns fp32 state_dict reconstructed from ds checkpoint
191
+
192
+ Args:
193
+ - ``ds_checkpoint_dir``: path to the deepspeed checkpoint folder (where the optimizer files are)
194
+
195
+ """
196
+ print(f"Processing zero checkpoint '{ds_checkpoint_dir}'")
197
+
198
+ optim_files = get_optim_files(ds_checkpoint_dir)
199
+ zero_stage, world_size, fp32_flat_groups = parse_optim_states(optim_files, ds_checkpoint_dir)
200
+ print(f"Detected checkpoint of type zero stage {zero_stage}, world_size: {world_size}")
201
+
202
+ model_files = get_model_state_files(ds_checkpoint_dir)
203
+
204
+ zero_model_states = parse_model_states(model_files)
205
+ print(f'Parsing checkpoint created by deepspeed=={zero_model_states[0].ds_version}')
206
+
207
+ if zero_stage <= 2:
208
+ return _get_fp32_state_dict_from_zero2_checkpoint(world_size, fp32_flat_groups, zero_model_states,
209
+ exclude_frozen_parameters)
210
+ elif zero_stage == 3:
211
+ return _get_fp32_state_dict_from_zero3_checkpoint(world_size, fp32_flat_groups, zero_model_states,
212
+ exclude_frozen_parameters)
213
+
214
+
215
+ def _zero2_merge_frozen_params(state_dict, zero_model_states):
216
+ if zero_model_states[0].frozen_param_shapes is None or len(zero_model_states[0].frozen_param_shapes) == 0:
217
+ return
218
+
219
+ frozen_param_shapes = zero_model_states[0].frozen_param_shapes
220
+ frozen_param_fragments = zero_model_states[0].frozen_param_fragments
221
+
222
+ if debug:
223
+ num_elem = sum(s.numel() for s in frozen_param_shapes.values())
224
+ print(f'rank 0: {FROZEN_PARAM_SHAPES}.numel = {num_elem}')
225
+
226
+ wanted_params = len(frozen_param_shapes)
227
+ wanted_numel = sum(s.numel() for s in frozen_param_shapes.values())
228
+ avail_numel = sum([p.numel() for p in frozen_param_fragments.values()])
229
+ print(f'Frozen params: Have {avail_numel} numels to process.')
230
+ print(f'Frozen params: Need {wanted_numel} numels in {wanted_params} params')
231
+
232
+ total_params = 0
233
+ total_numel = 0
234
+ for name, shape in frozen_param_shapes.items():
235
+ total_params += 1
236
+ unpartitioned_numel = shape.numel()
237
+ total_numel += unpartitioned_numel
238
+
239
+ state_dict[name] = frozen_param_fragments[name]
240
+
241
+ if debug:
242
+ print(f"{name} full shape: {shape} unpartitioned numel {unpartitioned_numel} ")
243
+
244
+ print(f"Reconstructed Frozen fp32 state dict with {total_params} params {total_numel} elements")
245
+
246
+
247
+ def _has_callable(obj, fn):
248
+ attr = getattr(obj, fn, None)
249
+ return callable(attr)
250
+
251
+
252
+ def _zero2_merge_trainable_params(state_dict, world_size, fp32_flat_groups, zero_model_states):
253
+ param_shapes = zero_model_states[0].param_shapes
254
+
255
+ # Reconstruction protocol:
256
+ #
257
+ # XXX: document this
258
+
259
+ if debug:
260
+ for i in range(world_size):
261
+ for j in range(len(fp32_flat_groups[0])):
262
+ print(f"{FP32_FLAT_GROUPS}[{i}][{j}].shape={fp32_flat_groups[i][j].shape}")
263
+
264
+ # XXX: memory usage doubles here (zero2)
265
+ num_param_groups = len(fp32_flat_groups[0])
266
+ merged_single_partition_of_fp32_groups = []
267
+ for i in range(num_param_groups):
268
+ merged_partitions = [sd[i] for sd in fp32_flat_groups]
269
+ full_single_fp32_vector = torch.cat(merged_partitions, 0)
270
+ merged_single_partition_of_fp32_groups.append(full_single_fp32_vector)
271
+ avail_numel = sum(
272
+ [full_single_fp32_vector.numel() for full_single_fp32_vector in merged_single_partition_of_fp32_groups])
273
+
274
+ if debug:
275
+ wanted_params = sum([len(shapes) for shapes in param_shapes])
276
+ wanted_numel = sum([sum(shape.numel() for shape in shapes.values()) for shapes in param_shapes])
277
+ # not asserting if there is a mismatch due to possible padding
278
+ print(f"Have {avail_numel} numels to process.")
279
+ print(f"Need {wanted_numel} numels in {wanted_params} params.")
280
+
281
+ # params
282
+ # XXX: for huge models that can't fit into the host's RAM we will have to recode this to support
283
+ # out-of-core computing solution
284
+ total_numel = 0
285
+ total_params = 0
286
+ for shapes, full_single_fp32_vector in zip(param_shapes, merged_single_partition_of_fp32_groups):
287
+ offset = 0
288
+ avail_numel = full_single_fp32_vector.numel()
289
+ for name, shape in shapes.items():
290
+
291
+ unpartitioned_numel = shape.numel() if _has_callable(shape, 'numel') else math.prod(shape)
292
+ total_numel += unpartitioned_numel
293
+ total_params += 1
294
+
295
+ if debug:
296
+ print(f"{name} full shape: {shape} unpartitioned numel {unpartitioned_numel} ")
297
+ state_dict[name] = full_single_fp32_vector.narrow(0, offset, unpartitioned_numel).view(shape)
298
+ offset += unpartitioned_numel
299
+
300
+ # Z2 started to align to 2*world_size to improve nccl performance. Therefore both offset and
301
+ # avail_numel can differ by anywhere between 0..2*world_size. Due to two unrelated complex
302
+ # paddings performed in the code it's almost impossible to predict the exact numbers w/o the
303
+ # live optimizer object, so we are checking that the numbers are within the right range
304
+ align_to = 2 * world_size
305
+
306
+ def zero2_align(x):
307
+ return align_to * math.ceil(x / align_to)
308
+
309
+ if debug:
310
+ print(f"original offset={offset}, avail_numel={avail_numel}")
311
+
312
+ offset = zero2_align(offset)
313
+ avail_numel = zero2_align(avail_numel)
314
+
315
+ if debug:
316
+ print(f"aligned offset={offset}, avail_numel={avail_numel}")
317
+
318
+ # Sanity check
319
+ if offset != avail_numel:
320
+ raise ValueError(f"consumed {offset} numels out of {avail_numel} - something is wrong")
321
+
322
+ print(f"Reconstructed fp32 state dict with {total_params} params {total_numel} elements")
323
+
324
+
325
+ def _get_fp32_state_dict_from_zero2_checkpoint(world_size, fp32_flat_groups, zero_model_states,
326
+ exclude_frozen_parameters):
327
+ state_dict = OrderedDict()
328
+
329
+ # buffers
330
+ buffers = zero_model_states[0].buffers
331
+ state_dict.update(buffers)
332
+ if debug:
333
+ print(f"added {len(buffers)} buffers")
334
+
335
+ if not exclude_frozen_parameters:
336
+ _zero2_merge_frozen_params(state_dict, zero_model_states)
337
+
338
+ _zero2_merge_trainable_params(state_dict, world_size, fp32_flat_groups, zero_model_states)
339
+
340
+ # recover shared parameters
341
+ for pair in zero_model_states[0].shared_params:
342
+ if pair[1] in state_dict:
343
+ state_dict[pair[0]] = state_dict[pair[1]]
344
+
345
+ return state_dict
346
+
347
+
348
+ def zero3_partitioned_param_info(unpartitioned_numel, world_size):
349
+ remainder = unpartitioned_numel % world_size
350
+ padding_numel = (world_size - remainder) if remainder else 0
351
+ partitioned_numel = math.ceil(unpartitioned_numel / world_size)
352
+ return partitioned_numel, padding_numel
353
+
354
+
355
+ def _zero3_merge_frozen_params(state_dict, world_size, zero_model_states):
356
+ if zero_model_states[0].frozen_param_shapes is None or len(zero_model_states[0].frozen_param_shapes) == 0:
357
+ return
358
+
359
+ if debug:
360
+ for i in range(world_size):
361
+ num_elem = sum(s.numel() for s in zero_model_states[i].frozen_param_fragments.values())
362
+ print(f'rank {i}: {FROZEN_PARAM_SHAPES}.numel = {num_elem}')
363
+
364
+ frozen_param_shapes = zero_model_states[0].frozen_param_shapes
365
+ wanted_params = len(frozen_param_shapes)
366
+ wanted_numel = sum(s.numel() for s in frozen_param_shapes.values())
367
+ avail_numel = sum([p.numel() for p in zero_model_states[0].frozen_param_fragments.values()]) * world_size
368
+ print(f'Frozen params: Have {avail_numel} numels to process.')
369
+ print(f'Frozen params: Need {wanted_numel} numels in {wanted_params} params')
370
+
371
+ total_params = 0
372
+ total_numel = 0
373
+ for name, shape in zero_model_states[0].frozen_param_shapes.items():
374
+ total_params += 1
375
+ unpartitioned_numel = shape.numel()
376
+ total_numel += unpartitioned_numel
377
+
378
+ param_frags = tuple(model_state.frozen_param_fragments[name] for model_state in zero_model_states)
379
+ state_dict[name] = torch.cat(param_frags, 0).narrow(0, 0, unpartitioned_numel).view(shape)
380
+
381
+ partitioned_numel, partitioned_padding_numel = zero3_partitioned_param_info(unpartitioned_numel, world_size)
382
+
383
+ if debug:
384
+ print(
385
+ f"Frozen params: {total_params} {name} full shape: {shape} partition0 numel={partitioned_numel} partitioned_padding_numel={partitioned_padding_numel}"
386
+ )
387
+
388
+ print(f"Reconstructed Frozen fp32 state dict with {total_params} params {total_numel} elements")
389
+
390
+
391
+ class GatheredTensor:
392
+ """
393
+ A pseudo tensor that collects partitioned weights.
394
+ It is more memory efficient when there are multiple groups.
395
+ """
396
+
397
+ def __init__(self, flat_groups, flat_groups_offset, offset, partitioned_numel, shape):
398
+ self.flat_groups = flat_groups
399
+ self.flat_groups_offset = flat_groups_offset
400
+ self.offset = offset
401
+ self.partitioned_numel = partitioned_numel
402
+ self.shape = shape
403
+ self.dtype = self.flat_groups[0][0].dtype
404
+
405
+ def contiguous(self):
406
+ """
407
+ Merge partitioned weights from flat_groups into a single tensor.
408
+ """
409
+ end_idx = self.offset + self.partitioned_numel
410
+ world_size = len(self.flat_groups)
411
+ pad_flat_param_chunks = []
412
+
413
+ for rank_i in range(world_size):
414
+ # for each rank, we need to collect weights from related group/groups
415
+ flat_groups_at_rank_i = self.flat_groups[rank_i]
416
+ start_group_id = None
417
+ end_group_id = None
418
+ for group_id in range(len(self.flat_groups_offset)):
419
+ if self.flat_groups_offset[group_id] <= self.offset < self.flat_groups_offset[group_id + 1]:
420
+ start_group_id = group_id
421
+ if self.flat_groups_offset[group_id] < end_idx <= self.flat_groups_offset[group_id + 1]:
422
+ end_group_id = group_id
423
+ break
424
+ # collect weights from related group/groups
425
+ for group_id in range(start_group_id, end_group_id + 1):
426
+ flat_tensor = flat_groups_at_rank_i[group_id]
427
+ start_offset = self.offset - self.flat_groups_offset[group_id]
428
+ end_offset = min(end_idx, self.flat_groups_offset[group_id + 1]) - self.flat_groups_offset[group_id]
429
+ pad_flat_param_chunks.append(flat_tensor[start_offset:end_offset])
430
+
431
+ # collect weights from all ranks
432
+ pad_flat_param = torch.cat(pad_flat_param_chunks, dim=0)
433
+ param = pad_flat_param[:self.shape.numel()].view(self.shape).contiguous()
434
+ return param
435
+
436
+
437
+ def _zero3_merge_trainable_params(state_dict, world_size, fp32_flat_groups, zero_model_states):
438
+ param_shapes = zero_model_states[0].param_shapes
439
+ avail_numel = sum([flat_group.numel() for flat_group in fp32_flat_groups[0]]) * world_size
440
+
441
+ # Reconstruction protocol: For zero3 we need to zip the partitions together at boundary of each
442
+ # param, re-consolidating each param, while dealing with padding if any
443
+
444
+ # merge list of dicts, preserving order
445
+ param_shapes = {k: v for d in param_shapes for k, v in d.items()}
446
+
447
+ if debug:
448
+ for i in range(world_size):
449
+ print(f"{FP32_FLAT_GROUPS}[{i}].shape={fp32_flat_groups[i].shape}")
450
+
451
+ wanted_params = len(param_shapes)
452
+ wanted_numel = sum(shape.numel() for shape in param_shapes.values())
453
+ # not asserting if there is a mismatch due to possible padding
454
+ avail_numel = fp32_flat_groups[0].numel() * world_size
455
+ print(f"Trainable params: Have {avail_numel} numels to process.")
456
+ print(f"Trainable params: Need {wanted_numel} numels in {wanted_params} params.")
457
+
458
+ # params
459
+ # XXX: for huge models that can't fit into the host's RAM we will have to recode this to support
460
+ # out-of-core computing solution
461
+ offset = 0
462
+ total_numel = 0
463
+ total_params = 0
464
+ flat_groups_offset = [0] + list(np.cumsum([flat_tensor.numel() for flat_tensor in fp32_flat_groups[0]]))
465
+ for name, shape in tqdm(param_shapes.items(), desc='Gathering sharded weights'):
466
+ unpartitioned_numel = shape.numel()
467
+ total_numel += unpartitioned_numel
468
+ total_params += 1
469
+ partitioned_numel, partitioned_padding_numel = zero3_partitioned_param_info(unpartitioned_numel, world_size)
470
+
471
+ if debug:
472
+ print(
473
+ f"Trainable params: {total_params} {name} full shape: {shape} partition0 numel={partitioned_numel} partitioned_padding_numel={partitioned_padding_numel}"
474
+ )
475
+
476
+ # memory efficient tensor
477
+ tensor = GatheredTensor(fp32_flat_groups, flat_groups_offset, offset, partitioned_numel, shape)
478
+ state_dict[name] = tensor
479
+ offset += partitioned_numel
480
+
481
+ offset *= world_size
482
+
483
+ # Sanity check
484
+ if offset != avail_numel:
485
+ raise ValueError(f"consumed {offset} numels out of {avail_numel} - something is wrong")
486
+
487
+ print(f"Reconstructed Trainable fp32 state dict with {total_params} params {total_numel} elements")
488
+
489
+
490
+ def _get_fp32_state_dict_from_zero3_checkpoint(world_size, fp32_flat_groups, zero_model_states,
491
+ exclude_frozen_parameters):
492
+ state_dict = OrderedDict()
493
+
494
+ # buffers
495
+ buffers = zero_model_states[0].buffers
496
+ state_dict.update(buffers)
497
+ if debug:
498
+ print(f"added {len(buffers)} buffers")
499
+
500
+ if not exclude_frozen_parameters:
501
+ _zero3_merge_frozen_params(state_dict, world_size, zero_model_states)
502
+
503
+ _zero3_merge_trainable_params(state_dict, world_size, fp32_flat_groups, zero_model_states)
504
+
505
+ # recover shared parameters
506
+ for pair in zero_model_states[0].shared_params:
507
+ if pair[1] in state_dict:
508
+ state_dict[pair[0]] = state_dict[pair[1]]
509
+
510
+ return state_dict
511
+
512
+
513
+ def to_torch_tensor(state_dict, return_empty_tensor=False):
514
+ """
515
+ Convert state_dict of GatheredTensor to torch tensor
516
+ """
517
+ torch_state_dict = {}
518
+ converted_tensors = {}
519
+ for name, tensor in state_dict.items():
520
+ tensor_id = id(tensor)
521
+ if tensor_id in converted_tensors: # shared tensors
522
+ shared_tensor = torch_state_dict[converted_tensors[tensor_id]]
523
+ torch_state_dict[name] = shared_tensor
524
+ else:
525
+ converted_tensors[tensor_id] = name
526
+ if return_empty_tensor:
527
+ torch_state_dict[name] = torch.empty(tensor.shape, dtype=tensor.dtype)
528
+ else:
529
+ torch_state_dict[name] = tensor.contiguous()
530
+ return torch_state_dict
531
+
532
+
533
+ def get_fp32_state_dict_from_zero_checkpoint(checkpoint_dir,
534
+ tag=None,
535
+ exclude_frozen_parameters=False,
536
+ lazy_mode=False):
537
+ """
538
+ Convert ZeRO 2 or 3 checkpoint into a single fp32 consolidated state_dict that can be loaded with
539
+ ``load_state_dict()`` and used for training without DeepSpeed or shared with others, for example
540
+ via a model hub.
541
+
542
+ Args:
543
+ - ``checkpoint_dir``: path to the desired checkpoint folder
544
+ - ``tag``: checkpoint tag used as a unique identifier for checkpoint. If not provided will attempt to load tag in 'latest' file. e.g., ``global_step14``
545
+ - ``exclude_frozen_parameters``: exclude frozen parameters
546
+ - ``lazy_mode``: get state_dict in lazy mode. It returns a dict of pesduo tensor instead of torch tensor, which is more memory efficient.
547
+ Convert the pesduo tensor to torch tensor by ``.contiguous()``
548
+
549
+ Returns:
550
+ - pytorch ``state_dict``
551
+
552
+ A typical usage might be ::
553
+
554
+ from deepspeed.utils.zero_to_fp32 import get_fp32_state_dict_from_zero_checkpoint
555
+ # do the training and checkpoint saving
556
+ state_dict = get_fp32_state_dict_from_zero_checkpoint(checkpoint_dir) # already on cpu
557
+ model = model.cpu() # move to cpu
558
+ model.load_state_dict(state_dict)
559
+ # submit to model hub or save the model to share with others
560
+
561
+ In this example the ``model`` will no longer be usable in the deepspeed context of the same
562
+ application. i.e. you will need to re-initialize the deepspeed engine, since
563
+ ``model.load_state_dict(state_dict)`` will remove all the deepspeed magic from it.
564
+
565
+ If you want it all done for you, use ``load_state_dict_from_zero_checkpoint`` instead.
566
+
567
+ Note: the above usage may not work if your application doesn't have sufficient free CPU memory.
568
+ You may need to use the offline approach using the ``zero_to_fp32.py`` script that is saved with
569
+ the checkpoint. Or you can load state_dict in lazy mode ::
570
+
571
+ from deepspeed.utils.zero_to_fp32 import get_fp32_state_dict_from_zero_checkpoint
572
+ state_dict = get_fp32_state_dict_from_zero_checkpoint(checkpoint_dir, lazy_mode=True) # not on cpu
573
+ for name, lazy_tensor in state_dict.item():
574
+ tensor = lazy_tensor.contiguous() # to cpu
575
+ print(name, tensor)
576
+ # del tensor to release memory if it no longer in use
577
+ """
578
+ if tag is None:
579
+ latest_path = os.path.join(checkpoint_dir, 'latest')
580
+ if os.path.isfile(latest_path):
581
+ with open(latest_path, 'r') as fd:
582
+ tag = fd.read().strip()
583
+ else:
584
+ raise ValueError(f"Unable to find 'latest' file at {latest_path}")
585
+
586
+ ds_checkpoint_dir = os.path.join(checkpoint_dir, tag)
587
+
588
+ if not os.path.isdir(ds_checkpoint_dir):
589
+ raise FileNotFoundError(f"Directory '{ds_checkpoint_dir}' doesn't exist")
590
+
591
+ state_dict = _get_fp32_state_dict_from_zero_checkpoint(ds_checkpoint_dir, exclude_frozen_parameters)
592
+ if lazy_mode:
593
+ return state_dict
594
+ else:
595
+ return to_torch_tensor(state_dict)
596
+
597
+
598
+ def convert_zero_checkpoint_to_fp32_state_dict(checkpoint_dir,
599
+ output_dir,
600
+ max_shard_size="5GB",
601
+ safe_serialization=False,
602
+ tag=None,
603
+ exclude_frozen_parameters=False):
604
+ """
605
+ Convert ZeRO 2 or 3 checkpoint into a single fp32 consolidated ``state_dict`` file that can be
606
+ loaded with ``torch.load(file)`` + ``load_state_dict()`` and used for training without DeepSpeed.
607
+
608
+ Args:
609
+ - ``checkpoint_dir``: path to the desired checkpoint folder. (one that contains the tag-folder, like ``global_step14``)
610
+ - ``output_dir``: directory to the pytorch fp32 state_dict output files
611
+ - ``max_shard_size``: the maximum size for a checkpoint before being sharded, default value is 5GB
612
+ - ``safe_serialization``: whether to save the model using `safetensors` or the traditional PyTorch way (that uses `pickle`).
613
+ - ``tag``: checkpoint tag used as a unique identifier for checkpoint. If not provided will attempt to load tag in the file named ``latest`` in the checkpoint folder, e.g., ``global_step14``
614
+ - ``exclude_frozen_parameters``: exclude frozen parameters
615
+ """
616
+
617
+ # Dependency pre-check
618
+ if safe_serialization:
619
+ try:
620
+ from safetensors.torch import save_file
621
+ except ImportError:
622
+ print('If you want to use `safe_serialization`, please `pip install safetensors`')
623
+ raise
624
+ if max_shard_size is not None:
625
+ try:
626
+ from huggingface_hub import split_torch_state_dict_into_shards
627
+ except ImportError:
628
+ print('If you want to use `max_shard_size`, please `pip install huggingface_hub`')
629
+ raise
630
+
631
+ # Convert zero checkpoint to state_dict
632
+ state_dict = get_fp32_state_dict_from_zero_checkpoint(checkpoint_dir,
633
+ tag,
634
+ exclude_frozen_parameters,
635
+ lazy_mode=True)
636
+
637
+ # Shard the model if it is too big.
638
+ weights_name = "model.safetensors" if safe_serialization else "pytorch_model.bin"
639
+ if max_shard_size is not None:
640
+ filename_pattern = weights_name.replace(".bin", "{suffix}.bin").replace(".safetensors", "{suffix}.safetensors")
641
+ # an memory-efficient approach for sharding
642
+ empty_state_dict = to_torch_tensor(state_dict, return_empty_tensor=True)
643
+ state_dict_split = split_torch_state_dict_into_shards(empty_state_dict,
644
+ filename_pattern=filename_pattern,
645
+ max_shard_size=max_shard_size)
646
+ else:
647
+ from collections import namedtuple
648
+ StateDictSplit = namedtuple("StateDictSplit", ["is_sharded", "filename_to_tensors"])
649
+ state_dict_split = StateDictSplit(is_sharded=False,
650
+ filename_to_tensors={weights_name: list(state_dict.keys())})
651
+
652
+ # Save the model by shard
653
+ os.makedirs(output_dir, exist_ok=True)
654
+ filename_to_tensors = state_dict_split.filename_to_tensors.items()
655
+ for shard_file, tensors in tqdm(filename_to_tensors, desc="Saving checkpoint shards"):
656
+ shard_state_dict = {tensor_name: state_dict[tensor_name] for tensor_name in tensors}
657
+ shard_state_dict = to_torch_tensor(shard_state_dict)
658
+ output_path = os.path.join(output_dir, shard_file)
659
+ if safe_serialization:
660
+ save_file(shard_state_dict, output_path, metadata={"format": "pt"})
661
+ else:
662
+ torch.save(shard_state_dict, output_path)
663
+ # release the memory of current shard
664
+ for tensor_name in list(shard_state_dict.keys()):
665
+ del state_dict[tensor_name]
666
+ del shard_state_dict[tensor_name]
667
+ del shard_state_dict
668
+ gc.collect()
669
+
670
+ # Save index if sharded
671
+ if state_dict_split.is_sharded:
672
+ index = {
673
+ "metadata": state_dict_split.metadata,
674
+ "weight_map": state_dict_split.tensor_to_filename,
675
+ }
676
+ save_index_file = "model.safetensors.index.json" if safe_serialization else "pytorch_model.bin.index.json"
677
+ save_index_file = os.path.join(output_dir, save_index_file)
678
+ with open(save_index_file, "w", encoding="utf-8") as f:
679
+ content = json.dumps(index, indent=2, sort_keys=True) + "\n"
680
+ f.write(content)
681
+
682
+
683
+ def load_state_dict_from_zero_checkpoint(model, checkpoint_dir, tag=None):
684
+ """
685
+ 1. Put the provided model to cpu
686
+ 2. Convert ZeRO 2 or 3 checkpoint into a single fp32 consolidated ``state_dict``
687
+ 3. Load it into the provided model
688
+
689
+ Args:
690
+ - ``model``: the model object to update
691
+ - ``checkpoint_dir``: path to the desired checkpoint folder. (one that contains the tag-folder, like ``global_step14``)
692
+ - ``tag``: checkpoint tag used as a unique identifier for checkpoint. If not provided will attempt to load tag in the file named ``latest`` in the checkpoint folder, e.g., ``global_step14``
693
+
694
+ Returns:
695
+ - ``model`: modified model
696
+
697
+ Make sure you have plenty of CPU memory available before you call this function. If you don't
698
+ have enough use the ``zero_to_fp32.py`` utility to do the conversion. You will find it
699
+ conveniently placed for you in the checkpoint folder.
700
+
701
+ A typical usage might be ::
702
+
703
+ from deepspeed.utils.zero_to_fp32 import load_state_dict_from_zero_checkpoint
704
+ model = load_state_dict_from_zero_checkpoint(trainer.model, checkpoint_dir)
705
+ # submit to model hub or save the model to share with others
706
+
707
+ Note, that once this was run, the ``model`` will no longer be usable in the deepspeed context
708
+ of the same application. i.e. you will need to re-initialize the deepspeed engine, since
709
+ ``model.load_state_dict(state_dict)`` will remove all the deepspeed magic from it.
710
+
711
+ """
712
+ logger.info(f"Extracting fp32 weights")
713
+ state_dict = get_fp32_state_dict_from_zero_checkpoint(checkpoint_dir, tag)
714
+
715
+ logger.info(f"Overwriting model with fp32 weights")
716
+ model = model.cpu()
717
+ model.load_state_dict(state_dict, strict=False)
718
+
719
+ return model
720
+
721
+
722
+ if __name__ == "__main__":
723
+ parser = argparse.ArgumentParser()
724
+ parser.add_argument("checkpoint_dir",
725
+ type=str,
726
+ help="path to the desired checkpoint folder, e.g., path/checkpoint-12")
727
+ parser.add_argument("output_dir",
728
+ type=str,
729
+ help="directory to the pytorch fp32 state_dict output files"
730
+ "(e.g. path/checkpoint-12-output/)")
731
+ parser.add_argument(
732
+ "--max_shard_size",
733
+ type=str,
734
+ default="5GB",
735
+ help="The maximum size for a checkpoint before being sharded. Checkpoints shard will then be each of size"
736
+ "lower than this size. If expressed as a string, needs to be digits followed by a unit (like `5MB`"
737
+ "We default it to 5GB in order for models to be able to run easily on free-tier google colab instances"
738
+ "without CPU OOM issues.")
739
+ parser.add_argument(
740
+ "--safe_serialization",
741
+ default=False,
742
+ action='store_true',
743
+ help="Whether to save the model using `safetensors` or the traditional PyTorch way (that uses `pickle`).")
744
+ parser.add_argument("-t",
745
+ "--tag",
746
+ type=str,
747
+ default=None,
748
+ help="checkpoint tag used as a unique identifier for checkpoint. e.g., global_step1")
749
+ parser.add_argument("--exclude_frozen_parameters", action='store_true', help="exclude frozen parameters")
750
+ parser.add_argument("-d", "--debug", action='store_true', help="enable debug")
751
+ args = parser.parse_args()
752
+
753
+ debug = args.debug
754
+
755
+ convert_zero_checkpoint_to_fp32_state_dict(args.checkpoint_dir,
756
+ args.output_dir,
757
+ max_shard_size=args.max_shard_size,
758
+ safe_serialization=args.safe_serialization,
759
+ tag=args.tag,
760
+ exclude_frozen_parameters=args.exclude_frozen_parameters)