Grunnleggende konsepter
Moreau Envelope-A (ME-A) is a novel algorithm that achieves uniform stability for adversarial training and weakly-convex non-smooth optimization problems, mitigating the issue of robust overfitting.
Sammendrag
The paper introduces Moreau Envelope-A (ME-A), a new algorithm designed to achieve uniform stability for adversarial training and weakly-convex non-smooth optimization problems.
Key highlights:
- Adversarial training suffers from the issue of robust overfitting, where the robust test accuracy decreases over epochs. This is attributed to the non-smoothness of the adversarial loss.
- Recent research has shown that the uniform stability bounds of stochastic gradient descent (SGD) for adversarial training include an additional term in O(T^qϵ), where T is the number of iterations and ϵ is the attack intensity. This term aligns with the observed robust overfitting.
- ME-A is introduced as a variant of the Moreau Envelope algorithm. It reformulates the original problem as a min-min problem, separating the non-strong convexity and non-smoothness. This allows ME-A to achieve uniform stability without additional computational overhead.
- ME-A is proven to achieve O(T^q/n)-uniform stability for both convex and weakly-convex non-smooth problems, where n is the number of training samples. This improves over SGD by reducing the O(T^qϵ) term.
- Experiments on SVHN, CIFAR-10, and CIFAR-100 datasets demonstrate that ME-A effectively mitigates the robust overfitting issue observed with SGD-based adversarial training.
- The paper also provides insights into the additive relationship between robust overfitting and sample complexity in adversarial training.
Statistikk
The robust test accuracy starts to decrease after a particular epoch in SGD-based adversarial training, while the robust training accuracy continues to increase.
The uniform stability bound of SGD for adversarial training includes an additional term in O(T^qϵ), which aligns with the observed robust overfitting.
ME-A reduces the O(T^qϵ) term in the uniform stability bound compared to SGD.
Sitater
"Recent research has utilized uniform stability, a generalization measure in learning theory, to investigate this phenomenon (Xing et al., 2021; Xiao et al., 2022b). They have suggested that the non-smoothness of the adversarial loss may contribute to the issue of robust overfitting."
"Consequently, the uniform stability bounds include an additional term in O(T^qϵ) (Xiao et al., 2022b), where ϵ is the attack intensity. The bound suggests that the robust test error increases as T grows, even when we have an infinite number of training samples (n →∞)."