FedNano: Toward Lightweight Federated Tuning for Pretrained Multimodal
  Large Language Models
Multimodal Large Language Models (MLLMs) excel in tasks like multimodal
reasoning and cross-modal retrieval but face deployment challenges in
real-world scenarios due to distributed multimodal data and strict privacy
requirements. Federated Learning (FL) offers a solution by enabling
collaborative model training without centralizing data. However, realizing FL
for MLLMs presents significant challenges, including high computational
demands, limited client capacity, substantial communication costs, and
heterogeneous client data. Existing FL methods assume client-side deployment of
full models, an assumption that breaks down for large-scale MLLMs due to their
massive size and communication demands. To address these limitations, we
propose FedNano, the first FL framework that centralizes the LLM on the server
while introducing NanoEdge, a lightweight module for client-specific
adaptation. NanoEdge employs modality-specific encoders, connectors, and
trainable NanoAdapters with low-rank adaptation. This design eliminates the
need to deploy LLM on clients, reducing client-side storage by 95%, and
limiting communication overhead to only 0.01% of the model parameters. By
transmitting only compact NanoAdapter updates, FedNano handles heterogeneous
client data and resource constraints while preserving privacy. Experiments
demonstrate that FedNano outperforms prior FL baselines, bridging the gap
between MLLM scale and FL feasibility, and enabling scalable, decentralized
multimodal AI systems.
6853c3af99bf39f9665c7a89
Yao Zhang
claimed_verified
2025-06-19T15:11:24.084Z
62cecf9c1415317d1fbf6cfe
/avatars/d30630ad96bcec1349728ba39476847a.svg
Yao Zhang
user
ZYao720
6853c3af99bf39f9665c7a8a
Hewei Gao
6853c3af99bf39f9665c7a8b
Haokun Chen
6853c3af99bf39f9665c7a8c
Weiguo Li
6853c3af99bf39f9665c7a8d
Yunpu Ma
6853c3af99bf39f9665c7a8e
Volker Tresp
FedNano: Toward Lightweight Federated Tuning for Pretrained Multimodal
  Large Language Models
Multimodal Large Language Models (MLLMs) excel in tasks like multimodal
reasoning and cross-modal retrieval but face deployment challenges in
real-world scenarios due to distributed multimodal data and strict privacy
requirements. Federated Learning (FL) offers a solution by enabling
collaborative model training without centralizing data. However, realizing FL
for MLLMs presents significant challenges, including high computational
demands, limited client capacity, substantial communication costs, and
heterogeneous client data. Existing FL methods assume client-side deployment of
full models, an assumption that breaks down for large-scale MLLMs due to their
massive size and communication demands. To address these limitations, we
propose FedNano, the first FL framework that centralizes the LLM on the server
while introducing NanoEdge, a lightweight module for client-specific
adaptation. NanoEdge employs modality-specific encoders, connectors, and
trainable NanoAdapters with low-rank adaptation. This design eliminates the
need to deploy LLM on clients, reducing client-side storage by 95%, and
limiting communication overhead to only 0.01% of the model parameters. By
transmitting only compact NanoAdapter updates, FedNano handles heterogeneous
client data and resource constraints while preserving privacy. Experiments
demonstrate that FedNano outperforms prior FL baselines, bridging the gap
between MLLM scale and FL feasibility, and enabling scalable, decentralized
multimodal AI systems.
FedNano is a federated learning framework that centralizes large language models on servers and uses NanoEdge modules for client-specific adaptation, addressing scalability and privacy issues.
Multimodal Large Language Models Federated Learning NanoEdge modality-specific encoders connectors NanoAdapters low-rank adaptation client-specific adaptation compact NanoAdapter updates decentralized multimodal AI systems