DeepSeek-V3.1-Terminus-W4AFP8

This model is a mixed-precision quantized version of DeepSeek-V3.1-Terminus, with dense layer keep the FP8 quantization of the original model, while MoE layers uses INT4 weights and FP8 activation, also called W4AFP8.

Benchmark

The accuracy below was obtained with SGLang V0.5.3 in non-thinking mode.

Model	math_500	gpqa	aime2024	mmlu-pro
DeepSeek-V3.1-Terminus-W4AFP8	89.83	78.28	80.0	83.66

Inference with SGLang

We have already supported deploying this model using tensor parallel in sglang for better performance. The releated PR https://github.com/sgl-project/sglang/pull/8118 has been merged in SGLang V0.5.2, so you can deploy this model using SGLang version 0.5.2 or later with tensor parallel.

python3 -m sglang.launch_server --model-path /path/to/DeepSeek-V3.1-Terminus-W4AFP8 --tp 8 --trust-remote-code --host 0.0.0.0 --port 8000