Sharpness-Aware Minimization: Analyzing the Edge of Stability and Its Impact on Neural Network Training

Sharpness-Aware Minimization (SAM) is a gradient-based neural network training algorithm that explicitly seeks to find solutions that avoid "sharp" minima. The authors derive an "edge of stability" for SAM, which depends on the norm of the gradient, and show empirically that SAM operates at this edge of stability across multiple deep learning tasks.