RSQ: Learning from Important Tokens Leads to Better Quantized LLMs
Layer-wise quantization is a key technique for efficiently compressing large
models without expensive retraining. Previous methods typically quantize the
weights of each layer by "uniformly" optimizing the layer reconstruction loss
across all output tokens. However, in this paper, we demonstrate that
better-quantized models can be obtained by prioritizing learning from important
tokens (e.g. which have large attention scores). Building on this finding, we
propose RSQ (Rotate, Scale, then Quantize), which (1) applies rotations
(orthogonal transformation) to the model to mitigate outliers (those with
exceptionally large magnitude), (2) scales the token feature based on its
importance, and (3) quantizes the model using the GPTQ framework with the
second-order statistics computed by scaled tokens. To compute token importance,
we explore both heuristic and dynamic strategies. Based on a thorough analysis
of all approaches, we adopt attention concentration, which uses attention
scores of each token as its importance, as the best approach. We demonstrate
that RSQ consistently outperforms baseline methods across multiple downstream
tasks and three model families: LLaMA3, Mistral, and Qwen2.5. Additionally,
models quantized with RSQ achieve superior performance on long-context tasks,
further highlighting its effectiveness. Lastly, RSQ demonstrates
generalizability across various setups, including different model sizes,
calibration datasets, bit precisions, and quantization methods.
67c7575d79b553252b95b0d6
Yi-Lin Sung
claimed_verified
2025-03-04T21:15:30.002Z
654ffe334d9e71e17becc660
/avatars/022b7a77051d26c4e5cbf254b7352eb9.svg
Yi-Lin Sung
user
a2889184
67c7575d79b553252b95b0d7
Prateek Yadav
67c7575d79b553252b95b0d8
Jialu Li
67c7575d79b553252b95b0d9
Jaehong Yoon
claimed_verified
2025-03-04T21:15:27.436Z
652066649004117947e46ed6
/avatars/972c97df6f26d2c3d6ce71ec579984bb.svg
Jaehong Yoon
user
jaehong31
67c7575d79b553252b95b0da
Mohit Bansal
RSQ: Learning from Important Tokens Leads to Better Quantized LLMs
Layer-wise quantization is a key technique for efficiently compressing large
models without expensive retraining. Previous methods typically quantize the
weights of each layer by "uniformly" optimizing the layer reconstruction loss
across all output tokens. However, in this paper, we demonstrate that
better-quantized models can be obtained by prioritizing learning from important
tokens (e.g. which have large attention scores). Building on this finding, we
propose RSQ (Rotate, Scale, then Quantize), which (1) applies rotations
(orthogonal transformation) to the model to mitigate outliers (those with
exceptionally large magnitude), (2) scales the token feature based on its
importance, and (3) quantizes the model using the GPTQ framework with the
second-order statistics computed by scaled tokens. To compute token importance,
we explore both heuristic and dynamic strategies. Based on a thorough analysis
of all approaches, we adopt attention concentration, which uses attention
scores of each token as its importance, as the best approach. We demonstrate
that RSQ consistently outperforms baseline methods across multiple downstream
tasks and three model families: LLaMA3, Mistral, and Qwen2.5. Additionally,
models quantized with RSQ achieve superior performance on long-context tasks,
further highlighting its effectiveness. Lastly, RSQ demonstrates
generalizability across various setups, including different model sizes,
calibration datasets, bit precisions, and quantization methods.
RSQ, a novel quantization technique that rotates, scales by importance, and then quantizes, improves model performance across various tasks and models by prioritizing tokens with large attention scores.
layer-wise quantization layer reconstruction loss token importance orthogonal transformation GPTQ framework attention concentration long-context tasks LLaMA3 Mistral Qwen2.5