The content discusses techniques to scale up the Banded Matrix Factorization (DP-BANDMF) mechanism, a differentially private machine learning algorithm that utilizes correlated noise to achieve better utility than the standard DP-SGD approach, especially in the large-epsilon, few-epoch regime.
The key challenges with DP-BANDMF are its computational and memory bottlenecks that limit its scalability to large-scale training scenarios. The paper presents two main innovations to address these limitations:
Efficient Strategy Optimization: The authors exploit the structure of banded strategies to reduce the per-iteration complexity of strategy optimization from O(n^3) to O(n^2b) time and O(nb) space, allowing DP-BANDMF to scale to approximately 10^5 training iterations. For even larger settings (up to 10^6 iterations), they restrict attention to banded Toeplitz strategies, whose structure enables O(n*b) time and O(n) space complexity during strategy optimization, with negligible loss in solution quality.
Distributed Noise Generation: Prior implementations of DP-MF-style mechanisms add noise on a single machine. The authors show how to effectively distribute the noise generation process, allowing DP-BANDMF to take advantage of multi-machine environments common in large-scale training regimes, and scale to larger models and more bands than was previously possible, with negligible training-time overhead.
Comprehensive experiments demonstrate that the authors' scalable DP-BANDMF mechanism offers lower expected error than all other scalable MF-style mechanisms, including DP-SGD and concurrent works, across a wide range of settings. The analysis also reveals that in many practical scenarios, the optimal number of bands is small, mitigating the memory overhead of DP-BANDMF compared to DP-SGD.
To Another Language
from source content
arxiv.org
Key Insights Distilled From
by Ryan McKenna at arxiv.org 10-01-2024
https://arxiv.org/pdf/2405.15913.pdfDeeper Inquiries