Adaptive Sigmoid Clipping for Balancing the Direction–Magnitude Mismatch Trade-off in Differentially Private Learning
Faeze Moradi Kalarde, Ali Bereyhi, Ben Liang, Min Dong
Differential privacy (DP) limits the impact of individual training data samples by clipping the sample gradients. Conventional clipping methods often assign unequal weights to the gradients, which prevents excessive magnitude attenuation after clipping. This however leads to a notable direction mismatch between the true batch gradient and the aggregation of the clipped gradients, causing training steps to move towards less desirable areas of the loss landscape. This work proposes a novel clipping method, termed adaptive sigmoid (AdaSig), which uses a sigmoid function with an adjustable saturation slope to clip the sample gradients. The slope is learned from data during the training phase to account for variations in sample gradient statistics over iterations. This enables the algorithm to control the direction mismatch while avoiding excessive reduction in the magnitude of the aggregation of the clipped gradients. Despite AdaSig's adaptive nature, our convergence analysis shows that DP-SGD with AdaSig clipping retains the best-known convergence rate in the non-convex loss setting. Evaluating AdaSig on sentence and image classification tasks across eight datasets shows that it consistently improves learning performance compared with established clipping methods.