Convergence Analysis of Continual Learning with Adaptive Gradient Methods

Continual learning can be formulated as a nonconvex finite-sum optimization problem, where the convergence of previous tasks and the current task are analyzed. Adaptive gradient methods are proposed to mitigate catastrophic forgetting by adjusting step sizes between tasks.