google/gemma-4-12B | INT4 (W4A16)
#20
by INC4AI - opened
Pipeline Failure Report
Model: google/gemma-4-12B
Quantization Scheme: INT4 (W4A16)
Failed Phase: quantize
Run ID: gemma-4-12B-AutoRound-W4A16-Tuning
Error Category: model_architecture_mismatch
Full Error Log
10:14:42 [INFO] HTTP Request: HEAD https://huggingface.co/datasets/NeelNanda/pile-10k/resolve/127bfedcd5047750df5ccf3a12979a47bfa0bafa/.huggingface.yaml "HTTP/1.1 404 Not Found"
10:14:42 [INFO] HTTP Request: GET https://datasets-server.huggingface.co/info?dataset=NeelNanda/pile-10k "HTTP/1.1 200 OK"
10:14:43 [INFO] HTTP Request: GET https://huggingface.co/api/datasets/NeelNanda/pile-10k/tree/127bfedcd5047750df5ccf3a12979a47bfa0bafa/data?recursive=true&expand=false "HTTP/1.1 200 OK"
10:14:43 [INFO] HTTP Request: GET https://huggingface.co/api/datasets/NeelNanda/pile-10k/tree/127bfedcd5047750df5ccf3a12979a47bfa0bafa?recursive=false&expand=false "HTTP/1.1 200 OK"
[33;1m2026-06-24 10:14:43 WARNING logging.py L340: Please note that 'shared_kv_states' key is not currently used in quantization fine-tuning.[0m
[38;20m2026-06-24 10:14:49 INFO data_driven.py L685: caching done[0m
0%| | 0/48 [00:00<?, ?it/s]
Quantizing model.language_model.layers.0: 0%| | 0/48 [00:01<?, ?it/s]/root/.venv/lib/python3.12/site-packages/torch/autograd/graph.py:869: UserWarning: Memory Efficient attention defaults to a non-deterministic algorithm. To explicitly enable determinism call torch.use_deterministic_algorithms(True, warn_only=False). (Triggered internally at /pytorch/aten/src/ATen/native/transformers/cuda/attention_backward.cu:900.)
return Variable._execution_engine.run_backward( # Calls into the C++ engine to run the backward pass
[38;20mquantized 7/7 layers in the block, loss iter 0: 0.053508 -> iter 175: 0.010552[0m
[38;20m2026-06-24 10:16:18 INFO device.py L1840: 'peak_ram': 10.6GB, 'peak_vram': 22.28GB[0m
Quantizing model.language_model.layers.1: 2%|▏ | 1/48 [01:29<1:09:48, 89.11s/it][38;20mquantized 7/7 layers in the block, loss iter 0: 0.016620 -> iter 197: 0.003855[0m
[38;20m2026-06-24 10:17:47 INFO device.py L1840: 'peak_ram': 11.43GB, 'peak_vram': 22.28GB[0m
Quantizing model.language_model.layers.1: 4%|▍ | 2/48 [02:57<1:08:09, 88.91s/it]
Quantizing model.language_model.layers.2: 4%|▍ | 2/48 [02:57<1:08:09, 88.91s/it]
Quantizing model.language_model.layers.2: 4%|▍ | 2/48 [03:10<1:08:09, 88.91s/it][38;20mquantized 7/7 layers in the block, loss iter 0: 0.006359 -> iter 167: 0.001274[0m
[38;20m2026-06-24 10:19:13 INFO device.py L1840: 'peak_ram': 12.35GB, 'peak_vram': 22.28GB[0m
Quantizing model.language_model.layers.3: 6%|▋ | 3/48 [04:24<1:06:40, 88.91s/it][38;20mquantized 7/7 layers in the block, loss iter 0: 0.008092 -> iter 186: 0.001709[0m
[38;20m2026-06-24 10:20:43 INFO device.py L1840: 'peak_ram': 13.12GB, 'peak_vram': 22.28GB[0m
Quantizing model.language_model.layers.3: 8%|▊ | 4/48 [05:54<1:04:56, 88.56s/it]
Quantizing model.language_model.layers.4: 8%|▊ | 4/48 [05:54<1:04:56, 88.56s/it]
Quantizing model.language_model.layers.4: 8%|▊ | 4/48 [06:10<1:04:56, 88.56s/it][38;20mquantized 7/7 layers in the block, loss iter 0: 0.008731 -> iter 194: 0.001733[0m
[38;20m2026-06-24 10:22:13 INFO device.py L1840: 'peak_ram': 13.98GB, 'peak_vram': 22.28GB[0m
Quantizing model.language_model.layers.5: 10%|█ | 5/48 [07:24<1:03:28, 88.56s/it]10:22:13 [ERROR] Quantization failed: The size of tensor a (512) must match the size of tensor b (256) at non-singleton dimension 3
Traceback (most recent call last):
File "/root/_work/1/s/auto_quant/phases/quantize.py", line 282, in <module>
quantize(args)
File "/root/_work/1/s/auto_quant/phases/quantize.py", line 183, in quantize
autoround.quantize()
File "/root/.venv/lib/python3.12/site-packages/auto_round/compressors/data_driven.py", line 722, in quantize
self._quantize_blocks(
File "/root/.venv/lib/python3.12/site-packages/auto_round/compressors/data_driven.py", line 529, in _quantize_blocks
reference_output = self.quantizer._get_block_outputs(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/root/.venv/lib/python3.12/site-packages/torch/utils/_contextlib.py", line 124, in decorate_context
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/root/.venv/lib/python3.12/site-packages/auto_round/algorithms/quantization/base.py", line 452, in _get_block_outputs
tmp_output = _bf(
^^^^
File "/root/.venv/lib/python3.12/site-packages/auto_round/compressors/utils.py", line 208, in block_forward
output = block(**input_others)
^^^^^^^^^^^^^^^^^^^^^
File "/root/.venv/lib/python3.12/site-packages/transformers/modeling_layers.py", line 93, in __call__
return super().__call__(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/root/.venv/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1779, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/root/.venv/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1790, in _call_impl
return forward_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/root/.venv/lib/python3.12/site-packages/transformers/models/gemma4_unified/modeling_gemma4_unified.py", line 516, in forward
hidden_states, _ = self.self_attn(
^^^^^^^^^^^^^^^
File "/root/.venv/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1779, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/root/.venv/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1790, in _call_impl
return forward_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/root/.venv/lib/python3.12/site-packages/transformers/models/gemma4_unified/modeling_gemma4_unified.py", line 421, in forward
query_states = apply_rotary_pos_emb(query_states, cos, sin, unsqueeze_dim=2)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/root/.venv/lib/python3.12/site-packages/transformers/models/gemma4_unified/modeling_gemma4_unified.py", line 304, in apply_rotary_pos_emb
return (x * cos) + (rotate_half(x) * sin)
~~^~~~~
RuntimeError: The size of tensor a (512) must match the size of tensor b (256) at non-singleton dimension 3
Quantizing model.language_model.layers.5: 10%|█ | 5/48 [07:24<1:03:44, 88.93s/it]
Auto-generated by error_analysis pipeline. cc @lvkaokao
In another discussion
lvkaokao changed discussion status to closed