FlyLoRA: Boosting Task Decoupling and Parameter Efficiency via Implicit Rank-Wise Mixture-of-Experts
Heming Zou, Yunliang Zang, Wutong Xu, Yao Zhu, Xiangyang Ji
Low-Rank Adaptation (LoRA) is a widely used parameter-efficient fine-tuning method for foundation models, but it suffers from parameter interference, resulting in suboptimal performance. Although Mixture-of-Experts (MoE)-based LoRA variants show promise in addressing this issue, they often struggle to balance decorrelation with computational efficiency. Inspired by the fly olfactory circuit, we propose FlyLoRA, an implicit MoE-based LoRA variant that introduces: (1) rank-wise expert activation in the up-projection matrix with implicit routing, and (2) frozen sparse random projection for the down-projection matrix, replacing the traditional dense updatable version. These components theoretically decouple intra- and inter-task interactions, enhancing decorrelation in both single-task instruction tuning and multi-task model fusion, respectively. FlyLoRA’s implicit routing and inherent sparsity further enhance computational efficiency. Extensive experiments across four domains --- general knowledge understanding, scientific question answering, mathematical reasoning, and code generation --- demonstrate consistent performance improvements over existing methods. Additionally, this work illustrates how biological structures can inspire innovations in AI technologies.