The author explores the importance of adaptive hyperparameter optimization in continual learning scenarios, emphasizing the need for dynamic tuning to improve performance and efficiency.
All the hyperparameter optimization (HPO) frameworks tested, including the commonly used but unrealistic end-of-training HPO, perform similarly in terms of predictive performance. The simplest and most computationally efficient method, first-task HPO, is recommended as the preferred HPO framework for continual learning.