
Removing the square root from adaptive gradient methods can close the generalization gap on convolutional architectures while maintaining performance on transformers, highlighting the overlooked role of adaptivity in their success.


coremsg

Analyzing-Adaptive-Gradient-Methods-Without-Square-Root


Analyzing Adaptive Gradient Methods Without Square-Root


title_rewrite


AdEMAMix, a novel optimizer, can leverage very old gradients to reach better solutions faster compared to the widely used Adam optimizer. This is achieved by combining a fast-changing and a slow-changing exponential moving average of gradients.


the-ademamix-optimizer-leveraging-old-gradients-for-faster-and-better-convergence-in-deep-learning


The AdEMAMix Optimizer: Leveraging Old Gradients for Faster and Better Convergence in Deep Learning
