FLiP: Towards Comprehensive and Reliable Evaluation of Federated Prompt Learning
Dongping Liao, Xitong Gao, Cheng-Zhong Xu
The increasing emphasis on privacy and data security has driven the adoption of federated learning (FL), a decentralized approach to train machine learning models without sharing raw data. Prompt learning (PL), which fine-tunes prompt embeddings of pretrained models, offers significant advantages in FL settings by reducing computational costs and communication overheads while leveraging the strong performance and generalization capabilities of vision-language models such as CLIP. To date, the reliable evaluation of federated prompt learning (FPL) algorithms for vision tasks remains an understudied area in current research. In this work, we introduce a comprehensive benchmark, named FLiP, to evaluate FPL algorithms. FLiP assesses the performance of 13 state-of-the-art centralized and FPL methods across 3 FL protocols and 12 open datasets, considering 6 distinct evaluation scenarios. Our findings demonstrate that PL maintains strong generalization performance in both in-distribution and out-of-distribution settings with minimal resource consumption. This work highlights the suitable application scenarios of each FPL algorithm in various environments characterized by data scarcity, unseen classes, and cross-domain distributional shifts. All benchmarks and code are publicly available to facilitate further research in this domain.