Federated Full-Parameter Tuning of Billion-Sized Language Models with
  Communication Cost under 18 Kilobytes
Pre-trained large language models (LLMs) require fine-tuning to improve their
responsiveness to natural language instructions. Federated learning (FL) offers
a way to perform fine-tuning using the abundant data on end devices without
compromising data privacy. Most existing federated fine-tuning methods for LLMs
rely on parameter-efficient fine-tuning techniques, which may not reach the
performance heights possible with full-parameter tuning. However, the
communication overhead associated with full-parameter tuning is prohibitively
high for both servers and clients. This work introduces FedKSeed, a novel
approach that employs zeroth-order optimization (ZOO) with a set of random
seeds. It enables federated full-parameter tuning of billion-sized LLMs
directly on devices. Our method significantly reduces transmission requirements
between the server and clients to just a few scalar gradients and random seeds,
amounting to only a few thousand bytes. Building on this, we develop a strategy
to assess the significance of ZOO perturbations for FL, allowing for
probability-differentiated seed sampling. This prioritizes perturbations that
have a greater impact on model accuracy. Experiments across six scenarios with
different LLMs, datasets and data partitions demonstrate that our approach
outperforms existing federated LLM fine-tuning methods in terms of both
communication efficiency and new task generalization.
6577f13432150025aca5762b
Zhen Qin
admin_assigned
2023-12-12T08:57:15.132Z
649014b91d71e55664838d2d
/avatars/f0e0f2830c5cb7428cbbc9634d95c34b.svg
Zhen Qin
user
zhenqincn
6577f13432150025aca5762c
Daoyuan Chen
admin_assigned
2023-12-12T08:57:31.994Z
6530ca70c1b8f265a431fc42
/avatars/7b1ec5b75398c8434ccd43cdce7fee53.svg
Daoyuan Chen
user
yxdyc
6577f13432150025aca5762d
Bingchen Qian
6577f13432150025aca5762e
Bolin Ding
6577f13432150025aca5762f
Yaliang Li
6577f13432150025aca57630
Shuiguang Deng
Federated Full-Parameter Tuning of Billion-Sized Language Models with
  Communication Cost under 18 Kilobytes
Pre-trained large language models (LLMs) require fine-tuning to improve their
responsiveness to natural language instructions. Federated learning (FL) offers
a way to perform fine-tuning using the abundant data on end devices without
compromising data privacy. Most existing federated fine-tuning methods for LLMs
rely on parameter-efficient fine-tuning techniques, which may not reach the
performance heights possible with full-parameter tuning. However, the
communication overhead associated with full-parameter tuning is prohibitively
high for both servers and clients. This work introduces FedKSeed, a novel
approach that employs zeroth-order optimization (ZOO) with a set of random
seeds. It enables federated full-parameter tuning of billion-sized LLMs
directly on devices. Our method significantly reduces transmission requirements
between the server and clients to just a few scalar gradients and random seeds,
amounting to only a few thousand bytes. Building on this, we develop a strategy
to assess the significance of ZOO perturbations for FL, allowing for
probability-differentiated seed sampling. This prioritizes perturbations that
have a greater impact on model accuracy. Experiments across six scenarios with
different LLMs, datasets and data partitions demonstrate that our approach
outperforms existing federated LLM fine-tuning methods in terms of both
communication efficiency and new task generalization.
FedKSeed uses zeroth-order optimization with random seeds to enable efficient, full-parameter federated fine-tuning of large language models on client devices.
pre-trained LLMs federated learning parameter-efficient fine-tuning zeroth-order optimization FedKSeed scalar gradients new task generalization