CrossSpectra: Exploiting Cross-Layer Smoothness for Parameter-Efficient Fine-Tuning
Yifei Zhang, Hao Zhu, Junhao Dong, Haoran Shi, Ziqiao Meng, Piotr Koniusz, Han Yu
Parameter-efficient fine-tuning (PEFT) methods are essential for adapting large foundation models to specific tasks without prohibitive storage costs. However, current approaches like LoRA treat each layer's adaptations independently, overlooking cross-layer relationships and resulting in parameter counts that scale linearly with model depth. We present a theoretical analysis showing that skip connections in transformers induce smooth gradient propagation across layers, leading to weight adaptations that naturally concentrate in low-frequency spectral components—particularly in the cross-layer dimension. Empirical analysis confirms this insight, revealing that nearly 70\% of adaptation energy concentrates in low frequencies. Building on this finding, we propose CrossSpectra, which parameterizes all attention weight adaptations (Q, K, V) across layers as a unified 3D tensor and decomposes it using sparse spectral coefficients. By employing $k_1$ non-zero coefficients within each layer's frequency space and truncating to $k_2$ frequencies across layers, CrossSpectra achieves parameter requirements of $O(k_1 k_2)$ compared to LoRA's $O(Lrd)$, where $L$ is the layer count and $r$ is the rank. Extensive experiments demonstrate that CrossSpectra matches or exceeds baseline performance while using 275× fewer parameters than LoRA—achieving 0.36\% of LoRA's parameters when fine-tuning LLaMA-7B on instruction-following tasks. Our approach shows that leveraging architectural properties through spectral analysis can yield dramatic efficiency gains in parameter-efficient fine-tuning.