mohammadmahdinouri
/

mol-base-init-final

Model card Files Files and versions

mohammadmahdinouri commited on Sep 14

Commit

28ab8d6

·

verified ·

1 Parent(s): 9017a1b

Upload DistillationWrapper

Files changed (1) hide show

config.json +1 -1

config.json CHANGED Viewed

@@ -15,7 +15,7 @@
   "initializer_range": 0.02,
   "intermediate_size": 2624,
   "layer_norm_eps": 1e-06,
-  "load_balancing_loss_coef": 0.2,
   "lora_alpha": 32,
   "lora_rank": 16,
   "max_position_embeddings": 8192,

   "initializer_range": 0.02,
   "intermediate_size": 2624,
   "layer_norm_eps": 1e-06,
+  "load_balancing_loss_coef": 0.15,
   "lora_alpha": 32,
   "lora_rank": 16,
   "max_position_embeddings": 8192,