Update README.md
Browse files
README.md
CHANGED
|
@@ -193,52 +193,3 @@ The training was done for 1 epochs We used 4 x [RTX 3090s](https://www.nvidia.c
|
|
| 193 |

|
| 194 |
|
| 195 |
|
| 196 |
-
optimizer: paged_adamw_8bit
|
| 197 |
-
lr_scheduler: constant_with_warmup
|
| 198 |
-
learning_rate: 1e-6
|
| 199 |
-
max_grad_norm: 0.01
|
| 200 |
-
|
| 201 |
-
train_on_inputs: false
|
| 202 |
-
group_by_length: false
|
| 203 |
-
bf16: auto
|
| 204 |
-
fp16:
|
| 205 |
-
tf32: true
|
| 206 |
-
|
| 207 |
-
gradient_checkpointing: unsloth
|
| 208 |
-
early_stopping_patience:
|
| 209 |
-
resume_from_checkpoint:
|
| 210 |
-
local_rank:
|
| 211 |
-
logging_steps: 1
|
| 212 |
-
xformers_attention:
|
| 213 |
-
flash_attention: true
|
| 214 |
-
|
| 215 |
-
warmup_steps: 25
|
| 216 |
-
evals_per_epoch: 4
|
| 217 |
-
eval_table_size:
|
| 218 |
-
eval_max_new_tokens: 128
|
| 219 |
-
saves_per_epoch: 1
|
| 220 |
-
debug:
|
| 221 |
-
deepspeed: /workspace/axolotl/deepspeed_configs/zero3_bf16_cpuoffload_params.json
|
| 222 |
-
weight_decay: 0.0
|
| 223 |
-
fsdp:
|
| 224 |
-
fsdp_config:
|
| 225 |
-
|
| 226 |
-
|
| 227 |
-
```
|
| 228 |
-
|
| 229 |
-
</details><br>
|
| 230 |
-
|
| 231 |
-
## Credits
|
| 232 |
-
|
| 233 |
-
Thank you to [Lucy Knada](https://huggingface.co/lucyknada), [Intervitens](https://huggingface.co/intervitens),[Cgato](https://huggingface.co/cgato), [Kubernetes Bad](https://huggingface.co/kubernetes-bad) and the rest of [Anthracite](https://huggingface.co/anthracite-org)
|
| 234 |
-
|
| 235 |
-
|
| 236 |
-
## Training
|
| 237 |
-
The training was done for 1 epochs We used 4 x [RTX 3090s](https://www.nvidia.com/en-us/geforce/graphics-cards/30-series/rtx-3090-3090ti/) GPUs graciously provided by [Intervitens](https://huggingface.co/intervitens) for the fine-tuning of the model.
|
| 238 |
-
|
| 239 |
-
[<img src="https://raw.githubusercontent.com/OpenAccess-AI-Collective/axolotl/main/image/axolotl-badge-web.png" alt="Built with Axolotl" width="200" height="32"/>](https://github.com/OpenAccess-AI-Collective/axolotl)
|
| 240 |
-
|
| 241 |
-
## Safety
|
| 242 |
-
|
| 243 |
-

|
| 244 |
-
|
|
|
|
| 193 |

|
| 194 |
|
| 195 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|