Bridging the Gap Between Promise and Performance for Microscaling FP4
  Quantization
The recent hardware-accelerated microscaling 4-bit floating-point formats
such as MXFP4 and NVFP4, supported on NVIDIA and AMD GPUs, promise to
revolutionize large language model (LLM) inference. Yet, their practical
benefits remain unproven. We present the first comprehensive study of MXFP4 and
NVFP4 for post-training quantization, revealing gaps between their promise and
real-world performance. Our analysis shows that state-of-the-art methods
struggle with FP4, due to two key issues: (1) NVFP4's small group size provably
neutralizes traditional outlier mitigation techniques; (2) MXFP4's power-of-two
scale quantization severely degrades accuracy due to high induced error. To
bridge this gap, we introduce Micro-Rotated-GPTQ (MR-GPTQ), a variant of the
classic GPTQ quantization algorithm that tailors the quantization process to
FP4's unique properties, by using block-wise Hadamard transforms and
format-specific optimizations. We support our proposal with a set of
high-performance GPU kernels that enable the MR-GPTQ format with negligible
overhead, by rotation fusion into the weights, and fast online computation of
the activations. This leads to speedups vs. FP16 of up to 3.6x layer-wise, and
2.2x end-to-end on NVIDIA B200, and of 6x layer-wise and 4x end-to-end on
RTX5090. Our extensive empirical evaluation demonstrates that MR-GPTQ matches
or outperforms state-of-the-art accuracy, significantly boosting MXFP4, to the
point where it nears that of NVFP4. We conclude that, while FP4 is not an
automatic upgrade over INT4, format-specialized methods like MR-GPTQ can unlock
a new frontier of accuracy-performance trade-offs.
68e3c1fb73e20ab5778421db
Vage Egiazarian
68e3c1fb73e20ab5778421dc
Roberto L. Castro
68e3c1fb73e20ab5778421dd
Denis Kuznedelev
claimed_verified
2025-10-07T12:28:16.063Z
629cf0475a13ba8233dd18c9
https://cdn-avatars.huggingface.co/v1/production/uploads/1654452258405-noauth.jpeg
Denis Kuznedelev
user
SpiridonSunRotator
68e3c1fb73e20ab5778421de
Andrei Panferov
68e3c1fb73e20ab5778421df
Eldar Kurtic
68e3c1fb73e20ab5778421e0
Shubhra Pandit
68e3c1fb73e20ab5778421e1
Alexandre Marques
68e3c1fb73e20ab5778421e2
Mark Kurtz
68e3c1fb73e20ab5778421e3
Saleh Ashkboos
68e3c1fb73e20ab5778421e4
Torsten Hoefler
68e3c1fb73e20ab5778421e5
Dan Alistarh
Bridging the Gap Between Promise and Performance for Microscaling FP4
  Quantization
The recent hardware-accelerated microscaling 4-bit floating-point formats
such as MXFP4 and NVFP4, supported on NVIDIA and AMD GPUs, promise to
revolutionize large language model (LLM) inference. Yet, their practical
benefits remain unproven. We present the first comprehensive study of MXFP4 and
NVFP4 for post-training quantization, revealing gaps between their promise and
real-world performance. Our analysis shows that state-of-the-art methods
struggle with FP4, due to two key issues: (1) NVFP4's small group size provably
neutralizes traditional outlier mitigation techniques; (2) MXFP4's power-of-two
scale quantization severely degrades accuracy due to high induced error. To
bridge this gap, we introduce Micro-Rotated-GPTQ (MR-GPTQ), a variant of the
classic GPTQ quantization algorithm that tailors the quantization process to
FP4's unique properties, by using block-wise Hadamard transforms and
format-specific optimizations. We support our proposal with a set of
high-performance GPU kernels that enable the MR-GPTQ format with negligible
overhead, by rotation fusion into the weights, and fast online computation of
the activations. This leads to speedups vs. FP16 of up to 3.6x layer-wise, and
2.2x end-to-end on NVIDIA B200, and of 6x layer-wise and 4x end-to-end on
RTX5090. Our extensive empirical evaluation demonstrates that MR-GPTQ matches
or outperforms state-of-the-art accuracy, significantly boosting MXFP4, to the
point where it nears that of NVFP4. We conclude that, while FP4 is not an
automatic upgrade over INT4, format-specialized methods like MR-GPTQ can unlock
a new frontier of accuracy-performance trade-offs.
A new quantization method, Micro-Rotated-GPTQ, addresses the challenges of 4-bit floating-point formats MXFP4 and NVFP4, achieving high performance and accuracy in large language model inference.
MXFP4 NVFP4 post-training quantization GPTQ block-wise Hadamard transforms format-specific optimizations GPU kernels rotation fusion online computation of activations accuracy-performance trade-offs