
This paper introduces a new class of adaptive learning rate optimizers, called Nlar optimizers, that dynamically estimate both learning rates and momentum using non-linear autoregressive time-series models, demonstrating robust convergence and strong initial adaptability compared to traditional methods like Adam.


coremsg

dynamic-estimation-of-learning-rates-using-a-non-linear-autoregressive-model-a-novel-approach-to-adaptive-learning-rate-optimization-in-machine-learning


Dynamic Estimation of Learning Rates Using a Non-Linear Autoregressive Model: A Novel Approach to Adaptive Learning Rate Optimization in Machine Learning


title_rewrite


Optimierung des adaptiven Lernraten für Follow-the-Regularized-Leader zur Erreichung optimaler Ergebnisse.


adaptive-learning-rate-for-follow-the-regularized-leader-competitive-ratio-analysis-and-best-of-both-worlds


Adaptive Learning Rate for Follow-the-Regularized-Leader: Competitive Ratio Analysis and Best-of-Both-Worlds



Prodigy is an algorithm that improves the convergence rate of D-Adaptation, providing a state-of-the-art solution for learning rate adaptation.


prodigy-an-adaptive-learning-rate-algorithm-for-optimization


Prodigy: An Adaptive Learning Rate Algorithm for Optimization
