any4: Learned 4-bit Numeric Representation for LLMs
We present any4, a learned 4-bit weight quantization solution for large
language models (LLMs) providing arbitrary numeric representations without
requiring pre-processing of weights or activations. any4 yields higher accuracy
compared to other related 4-bit numeric representation types: int4, fp4 and
nf4, as evaluated on a range of model sizes, generations and families (Llama 2,
Llama 3, Mistral and Mixtral). While any4 does not require preprocessing of
weights or activations, it is also competitive with orthogonal techniques that
require such preprocessing (e.g., AWQ and GPTQ). We also experiment with any3
and any2 and show competitiveness at lower bits. Additionally, we show that we
can calibrate using a single curated diverse sample rather than hundreds of
samples from a dataset as done in most quantization approaches. We also open
source tinygemm, a latency optimized GPU matrix multiplication library for
LLMs, that implements any4 using a GPU-efficient lookup table strategy along
with other common quantization methods. We open source our code at
https://github.com/facebookresearch/any4 .
686dcc0bcb5725779c60b38f
Mostafa Elhoushi
claimed_verified
2025-07-09T08:50:25.543Z
63c9725ebedad7e2bf160bdc
https://cdn-avatars.huggingface.co/v1/production/uploads/63c9725ebedad7e2bf160bdc/wzPuyhOXCYBNGwZDshbnL.jpeg
Mostafa Elhoushi
user
melhoushi
686dcc0bcb5725779c60b390
Jeff Johnson
any4: Learned 4-bit Numeric Representation for LLMs
We present any4, a learned 4-bit weight quantization solution for large
language models (LLMs) providing arbitrary numeric representations without
requiring pre-processing of weights or activations. any4 yields higher accuracy
compared to other related 4-bit numeric representation types: int4, fp4 and
nf4, as evaluated on a range of model sizes, generations and families (Llama 2,
Llama 3, Mistral and Mixtral). While any4 does not require preprocessing of
weights or activations, it is also competitive with orthogonal techniques that
require such preprocessing (e.g., AWQ and GPTQ). We also experiment with any3
and any2 and show competitiveness at lower bits. Additionally, we show that we
can calibrate using a single curated diverse sample rather than hundreds of
samples from a dataset as done in most quantization approaches. We also open
source tinygemm, a latency optimized GPU matrix multiplication library for
LLMs, that implements any4 using a GPU-efficient lookup table strategy along
with other common quantization methods. We open source our code at
https://github.com/facebookresearch/any4 .
any4 is a learned 4-bit weight quantization method for large language models that achieves high accuracy without preprocessing and uses a GPU-efficient lookup table strategy.
4-bit weight quantization LLMs int4 fp4 nf4 AWQ GPTQ any3 any2 latency optimized GPU matrix multiplication lookup table strategy