Tight Convergence Rate of RMSProp and Its Momentum Extension Measured by ℓ1 Norm
The paper establishes the convergence rate of 1/T ∑T k=1 E[‖∇f(xk)‖1] ≤ Õ(√d/T^(1/4)) for RMSProp and its momentum extension, measured by ℓ1 norm, without the bounded gradient assumption.
An Enhanced Grey Wolf Optimizer with Elite Inheritance and Balance Search Mechanisms for Improved Optimization Performance
The proposed EBGWO algorithm incorporates an elite inheritance mechanism and a balance search mechanism to improve the convergence effect and balance exploration and exploitation capabilities of the original Grey Wolf Optimizer (GWO) algorithm.
Implicit Bias of AdamW: Constrained Optimization under ℓ∞ Norm
AdamW implicitly performs constrained optimization under the ℓ∞ norm constraint, converging to the KKT points of the constrained problem.
Derivative-free Tree Optimization for High-Dimensional Complex Systems
A novel derivative-free stochastic tree search (DOTS) method that enables accelerated optimal design of high-dimensional complex systems by constructing a stochastic search tree with a short-range backpropagation mechanism and a dynamic upper confidence bound.
Parametric-Task MAP-Elites: A Black-Box Algorithm for Continuous Multi-Task Optimization
Parametric-Task MAP-Elites (PT-ME) is a new black-box algorithm that efficiently solves continuous multi-task optimization problems by covering the task parameter space with high-quality solutions.
Theoretical and Empirical Analysis of Adam Optimizer with Constant Step Size in Non-Convex Settings
This work provides theoretical guarantees for the convergence of the Adam optimizer with a constant step size in non-convex settings. It also proposes a method to estimate the Lipschitz constant of the loss function, which is crucial for determining the optimal constant step size.
Convergence Analysis of RMSProp and Adam Optimizers for Generalized-smooth Non-convex Optimization with Affine Noise Variance
This paper provides the first tight convergence analyses for RMSProp and Adam optimizers in non-convex optimization under the most relaxed assumptions of coordinate-wise generalized smoothness and affine noise variance.
Discrete Natural Evolution Strategies: Optimizing Discrete Parameters without Gradients
Natural evolution strategies can be extended to discrete parameter spaces, allowing optimization of models with discrete parameters without the need for gradient computation.
Nonlinear African Vulture Optimization Algorithm with Henon Chaotic Mapping and Reverse Learning
Enhancing the African Vulture Optimization Algorithm through innovative strategies.