Accelerating Differentially Private Fine-Tuning with Momentum and Optimal Hyperparameter Scaling

By carefully integrating techniques from prior work, including momentum acceleration and a new linear scaling rule for hyperparameters, we obtain new state-of-the-art performance on benchmark computer vision and natural language processing tasks under differential privacy constraints.