The key highlights and insights of the content are:
The authors introduce R2N, a modified quasi-Newton method for solving nonsmooth regularized optimization problems of the form min_x f(x) + h(x), where f is C1 and h is lower semi-continuous and prox-bounded. Both f and h may be nonconvex.
At each iteration, R2N computes a step by minimizing the sum of a quadratic model of f, a model of h, and an adaptive quadratic regularization term. The Hessian of the quadratic model of f may be the true Hessian or an approximation such as a quasi-Newton update.
The authors establish global convergence of R2N under the assumption that the models of h approximate h(x + s) as o(||s||), which covers composite terms with Hölder continuous Jacobians. No assumption on local Lipschitz continuity of ∇f or boundedness of the model Hessians is required, as long as they do not diverge too fast.
The authors also develop R2DH, a variant of R2N where the model Hessian is diagonal, which allows for explicit computation of the step in certain cases without the need for a subproblem solver.
Complexity results are derived for R2N and R2DH, accounting for potentially unbounded model Hessians. Tight bounds of O(ε^(-2/(1-p))) and O(exp(cε^(-2))) are established, where p controls the growth of the model Hessians.
The authors provide efficient implementations of R2N and R2DH in Julia and demonstrate their performance on challenging problems, including minimum-rank matrix completion, which cannot be solved directly by trust-region methods.
To Another Language
from source content
arxiv.org
Key Insights Distilled From
by Youssef Diou... at arxiv.org 10-01-2024
https://arxiv.org/pdf/2409.19428.pdfDeeper Inquiries