QWHA: Quantization-Aware Walsh-Hadamard Adaptation for
  Parameter-Efficient Fine-Tuning on Large Language Models
The demand for efficient deployment of large language models (LLMs) has
driven interest in quantization, which reduces inference cost, and
parameter-efficient fine-tuning (PEFT), which lowers training overhead. This
motivated the development of quantization-aware PEFT to produce accurate yet
efficient quantized models. In this setting, reducing quantization error prior
to fine-tuning is crucial for achieving high model accuracy. However, existing
methods that rely on low-rank adaptation suffer from limited representational
capacity. Recent Fourier-related transform (FT)-based adapters offer greater
representational power than low-rank adapters, but their direct integration
into quantized models often results in ineffective error reduction and
increased computational overhead. To overcome these limitations, we propose
QWHA, a method that integrates FT-based adapters into quantized models by
employing the Walsh-Hadamard Transform (WHT) as the transform kernel, together
with a novel adapter initialization scheme incorporating adaptive parameter
selection and value refinement. We demonstrate that QWHA effectively mitigates
quantization errors while facilitating fine-tuning, and that its design
substantially reduces computational cost. Experimental results show that QWHA
consistently outperforms baselines in low-bit quantization accuracy and
achieves significant training speedups over existing FT-based adapters. The
code is available at https://github.com/vantaa89/qwha.
68d24c461ca7156988a8eeae
Hyesung Jeon
admin_assigned
2025-09-23T14:27:46.208Z
6400208acafc9d549863af59
/avatars/6c383c810a038ce61e803f1d75132471.svg
Hyesung Jeon
user
hjeon2k
68d24c461ca7156988a8eeaf
Seojune Lee
admin_assigned
2025-09-23T14:27:53.916Z
655f207afcbe7329b9502b56
/avatars/521c76b785336fb9d015bb4442b58475.svg
Seojune Lee
user
vantaa32
68d24c461ca7156988a8eeb0
Beomseok Kang
admin_assigned
2025-09-23T14:28:01.608Z
66734e27a0ea9fb0d7d06e68
/avatars/4c7f2e28af85cd1baf2abcc2280c4835.svg
Beom Seok Kang
user
beomseok-kang
68d24c461ca7156988a8eeb1
Yulhwa Kim
68d24c461ca7156988a8eeb2
Jae-Joon Kim
QWHA: Quantization-Aware Walsh-Hadamard Adaptation for
  Parameter-Efficient Fine-Tuning on Large Language Models
The demand for efficient deployment of large language models (LLMs) has
driven interest in quantization, which reduces inference cost, and
parameter-efficient fine-tuning (PEFT), which lowers training overhead. This
motivated the development of quantization-aware PEFT to produce accurate yet
efficient quantized models. In this setting, reducing quantization error prior
to fine-tuning is crucial for achieving high model accuracy. However, existing
methods that rely on low-rank adaptation suffer from limited representational
capacity. Recent Fourier-related transform (FT)-based adapters offer greater
representational power than low-rank adapters, but their direct integration
into quantized models often results in ineffective error reduction and
increased computational overhead. To overcome these limitations, we propose
QWHA, a method that integrates FT-based adapters into quantized models by
employing the Walsh-Hadamard Transform (WHT) as the transform kernel, together
with a novel adapter initialization scheme incorporating adaptive parameter
selection and value refinement. We demonstrate that QWHA effectively mitigates
quantization errors while facilitating fine-tuning, and that its design
substantially reduces computational cost. Experimental results show that QWHA
consistently outperforms baselines in low-bit quantization accuracy and
achieves significant training speedups over existing FT-based adapters. The
code is available at https://github.com/vantaa89/qwha.
QWHA integrates Walsh-Hadamard Transform-based adapters into quantized models to reduce quantization errors and computational overhead, improving low-bit quantization accuracy and training speed.
quantization parameter-efficient fine-tuning quantization-aware PEFT quantization error low-rank adaptation Fourier-related transform FT-based adapters Walsh-Hadamard Transform WHT adaptive parameter selection value refinement low-bit quantization accuracy training speedups