Cannot load w/ sglang 0.5.6
#1
by
huggyfaceenjoyer
- opened
AttributeError: 'Glm4vMoeConfig' object has no attribute 'rope_scaling'
4.6v
"rope_parameters": {
"mrope_section": [
8,
12,
12
],
"partial_rotary_factor": 0.5,
"rope_theta": 500000,
"rope_type": "default"
},
4.5 Air
"rope_scaling": null,
"rope_theta": 1000000,
use sglang 0.5.6.post1
use sglang 0.5.6.post1
That got me to be able to get further along, but I run out of vram with same settings as Air FP8 with 4.6V FP8, even drastically reducing context window to minimum. Dual RTX 6000 Pro 96G.
Also having a lot of problems with 0.5.6.post1 and/or prerelease transformers where even GLM 4.5 Air is no longer stable and had to revert.
Any update on this, Did you manage to get this model running?
Any update on this, Did you manage to get this model running?
No, I stopped trying as I was more focused on M2.1.