Efficient Optimization on the Random Generalized Stiefel Manifold without Retraction

The authors propose a cheap stochastic iterative method that solves the optimization problem on the random generalized Stiefel manifold without requiring expensive eigenvalue decompositions or matrix inversions. The method has lower per-iteration cost, requires only matrix multiplications, and has the same convergence rates as its Riemannian counterparts.