featherless-ai
/

QRWKV-72B

Text Generation

Model card Files Files and versions

SmerkyG commited on Sep 16

Commit

85678d7

·

verified ·

1 Parent(s): 7598ea3

Update config.json

Files changed (1) hide show

config.json +1 -1

config.json CHANGED Viewed

@@ -15,7 +15,7 @@
   "hidden_size": 8192,
   "initializer_range": 0.02,
   "intermediate_size": 29568,
-  "lora_rank_tokenshift": 64,
   "lora_rank_decay": 128,
   "max_position_embeddings": 131072,
   "max_window_layers": 80,

   "hidden_size": 8192,
   "initializer_range": 0.02,
   "intermediate_size": 29568,
+  "lora_rank_tokenshift": 160,
   "lora_rank_decay": 128,
   "max_position_embeddings": 131072,
   "max_window_layers": 80,