steampunque/gemma-3-12b-it-MP-GGUF

Mixed Precision GGUF layer quantization of gemma-3-12b-it by Google

Original model: https://huggingface.co/google/gemma-3-12b-it

The hybrid quant employs different quantization levels on a per layer basis to increased flexibility of trading off performance vs file size. Less parameter bits are used at deep layers and more bits at cortex layers to simultaneously optimize quantized size and model performance. An extended layer definition E quant Q4_E_H for the model is defined as follows:

   LAYER_TYPES='[
   ["A","attn","Q","attn_q","K","attn_k","V","attn_v","O","attn_o","S","ssm","F","ffn","G","ffn_g","U","ffn_u","D","ffn_d"],
   ["MAP","VOD","0","QN_K","2","Q2_K","3","Q3_K","4","Q4_K","5","Q5_K","6","Q6_K","8","Q8_0","h","F16","f","F32"],
   [0 ,"Q6_K_666"],[1 ,"Q6_K_655"],[2 ,"Q5_K_655"],[3 ,"Q5_K_555"],[4 ,"Q4_K_544"],[5 ,"Q4_K_666"],[6 ,"Q4_K_544"],[7 ,"Q4_K_554"],
   [8 ,"Q4_K_544"],[9 ,"Q4_K_544"],[10,"Q4_K_544"],[11,"Q4_K_666"],[12,"Q4_K_544"],[13,"Q4_K_544"],[14,"Q4_K_544"],[15,"Q4_K_544"],
   [16,"Q4_K_544"],[17,"Q4_K_555"],[18,"Q4_K_544"],[19,"Q4_K_544"],[20,"Q4_K_544"],[21,"Q4_K_544"],[22,"Q4_K_544"],[23,"Q4_K_555"],
   [24,"Q4_K_644"],[25,"Q4_K_644"],[26,"Q4_K_644"],[27,"Q4_K_644"],[28,"Q4_K_644"],[29,"Q4_K_655"],[30,"Q4_K_644"],[31,"Q4_K_644"],
   [32,"Q4_K_654"],[33,"Q4_K_654"],[34,"Q4_K_654"],[35,"Q4_K_655"],[36,"Q4_K_654"],[37,"Q4_K_654"],[38,"Q4_K_654"],[39,"Q4_K_654"],
   [40,"Q4_K_654"],[41,"Q5_K_666"],[42,"Q4_K_665"],[43,"Q4_K_666"],[44,"Q5_K_666"],[45,"Q5_K_666"],[46,"Q5_K_668"],[47,"Q6_K_886"]
   ]'
   FLAGS="--token-embedding-type Q6_K --output-tensor-type Q6_K --layer-types-high"

The quant was optimized for high reasoning + knowledge performance across a range of test prompts while sized to approximate Q4_K_M bpw.

Comparison:

Quant	size	PPL	Comment
Q4_K_M	7.3e9	-	default embed and output
Q4_E_H	7.5e9	9.1	Q6_K embed Q6_K output

Usage:

gemma-3 12b is a vision capable model. It can be used together with its multimedia projector layers to process images and text inputs and generate text outputs. The mmproj file is made available in this repository. To test vision mode follow the docs in the mtmd readme in the tools directory of the source tree https://github.com/ggml-org/llama.cpp/blob/master/tools/mtmd/README.md .

The model also uses sliding window attention. Use of llama.cpp b5554 and above is recommend for support of the SWA mode. If --swa-full flag is used, the old method of keeping all KV memory and masking out everything outside the SWA window is used.

Running:

The model can be speculated with gemma 3 270m it. Approximate performance using a downstream llama.cpp server with cuda backend and custome speculation code on a RTX 4070 (b9940 downstream):

ND	QKV	NKV	PP	TG
4	F16	50k	1000	86
0	F16	60k	1000	53
4	Q8_0	82k	1000	82
0	Q8_0	95k	1000	52

The model passes a basic long context test but will not correctly solve the long runescape prompt https://thireus.com/REDDIT/Qwen3_Runescape_Massive_Prompt.txt trimmed back to fit in available context. This prompt was most likely not available during pretrain of gemma 3.

Vision:

The model correctly identifies two tough bird-id images which gemma-4-12b-it fails, although gemma-4 does score higher on synthetic vision evals.

Code:

The model is capable of creating working code on simple prompts but failed about half the test cases given.

Benchmarks:

Benchmarks for the model (vision only) are given here: https://huggingface.co/spaces/steampunque/benchlm

Download the file from below:

Link	Type	Size/e9 B	Notes
gemma-3-12b-it.Q4_E_H.gguf	Q4_E_H	7.5e9 B	~Q4_K_M size
gemma-3-12b-it.mmproj.gguf	mmproj	0.85e9 B	multimedia projector

A discussion thread about the hybrid layer quant approach can be found here on the llama.cpp git repository:

https://github.com/ggml-org/llama.cpp/discussions/13040