Efficient Differentially Private Fine-Tuning of LLMs via Reinforcement
  Learning
The tension between data privacy and model utility has become the defining
bottleneck for the practical deployment of large language models (LLMs) trained
on sensitive corpora including healthcare. Differentially private stochastic
gradient descent (DP-SGD) guarantees formal privacy, yet it does so at a
pronounced cost: gradients are forcibly clipped and perturbed with noise,
degrading sample efficiency and final accuracy. Numerous variants have been
proposed to soften this trade-off, but they all share a handicap: their control
knobs are hard-coded, global, and oblivious to the evolving optimization
landscape. Consequently, practitioners are forced either to over-spend privacy
budget in pursuit of utility, or to accept mediocre models in order to stay
within privacy constraints. We present RLDP, the first framework to cast DP
optimization itself as a closed-loop control problem amenable to modern deep
reinforcement learning (RL). RLDP continuously senses rich statistics of the
learning dynamics and acts by selecting fine-grained per parameter
gradient-clipping thresholds as well as the magnitude of injected Gaussian
noise. A soft actor-critic (SAC) hyper-policy is trained online during language
model fine-tuning; it learns, from scratch, how to allocate the privacy budget
where it matters and when it matters. Across more than 1,600 ablation
experiments on GPT2-small, Llama-1B, Llama-3B, and Mistral-7B, RLDP delivers
perplexity reductions of 1.3-30.5% (mean 5.4%) and an average 5.6% downstream
utility gain. RLDP reaches each baseline's final utility after only 13-43% of
the gradient-update budget (mean speed-up 71%), all while honoring the same
(epsilon, delta)-DP contract and exhibiting equal or lower susceptibility
to membership-inference and canary-extraction attacks.
688b0f7c8b724c8c7187de5f
Afshin Khadangi
claimed_verified
2025-07-31T08:14:44.109Z
632292053007fcbf2932da22
https://cdn-avatars.huggingface.co/v1/production/uploads/632292053007fcbf2932da22/Cf4w-OunyQzHAJhyZsHTf.jpeg
Afshin Khadangi
user
akhadangi
688b0f7c8b724c8c7187de60
Amir Sartipi
claimed_verified
2025-07-31T21:00:53.834Z
6698e7cbed81d2c9bc58f784
/avatars/c2fbf808488ef4e157ec9627560a7480.svg
Amir Sartipi
user
asartipi
688b0f7c8b724c8c7187de61
Igor Tchappi
688b0f7c8b724c8c7187de62
Ramin Bahmani
688b0f7c8b724c8c7187de63
Gilbert Fridgen
Efficient Differentially Private Fine-Tuning of LLMs via Reinforcement
  Learning
The tension between data privacy and model utility has become the defining
bottleneck for the practical deployment of large language models (LLMs) trained
on sensitive corpora including healthcare. Differentially private stochastic
gradient descent (DP-SGD) guarantees formal privacy, yet it does so at a
pronounced cost: gradients are forcibly clipped and perturbed with noise,
degrading sample efficiency and final accuracy. Numerous variants have been
proposed to soften this trade-off, but they all share a handicap: their control
knobs are hard-coded, global, and oblivious to the evolving optimization
landscape. Consequently, practitioners are forced either to over-spend privacy
budget in pursuit of utility, or to accept mediocre models in order to stay
within privacy constraints. We present RLDP, the first framework to cast DP
optimization itself as a closed-loop control problem amenable to modern deep
reinforcement learning (RL). RLDP continuously senses rich statistics of the
learning dynamics and acts by selecting fine-grained per parameter
gradient-clipping thresholds as well as the magnitude of injected Gaussian
noise. A soft actor-critic (SAC) hyper-policy is trained online during language
model fine-tuning; it learns, from scratch, how to allocate the privacy budget
where it matters and when it matters. Across more than 1,600 ablation
experiments on GPT2-small, Llama-1B, Llama-3B, and Mistral-7B, RLDP delivers
perplexity reductions of 1.3-30.5% (mean 5.4%) and an average 5.6% downstream
utility gain. RLDP reaches each baseline's final utility after only 13-43% of
the gradient-update budget (mean speed-up 71%), all while honoring the same
(epsilon, delta)-DP contract and exhibiting equal or lower susceptibility
to membership-inference and canary-extraction attacks.
RLDP, a deep reinforcement learning framework, optimizes differentially private training by dynamically adjusting gradient clipping and noise, enhancing model utility and speed while maintaining privacy.
differentially private stochastic gradient descent DP-SGD deep reinforcement learning RL soft actor-critic SAC gradient-clipping thresholds Gaussian noise ($\epsilon$ $\delta$)-DP membership-inference canary-extraction attacks