Core Concepts
Neural networks can be converted to function space through a dual parameterization, enabling efficient sequential learning.
Abstract
The content discusses the challenges of gradient-based deep learning in sequential learning paradigms and introduces a technique to convert neural networks from weight space to function space. It highlights the benefits of this conversion, such as scalability, retention of prior knowledge, and efficient incorporation of new data without retraining. The paper also compares this approach with Gaussian processes and other methods in various experiments.
Abstract:
Challenges in gradient-based deep learning for sequential learning.
Introduction of a technique converting neural networks to function space.
Benefits include scalability, retention of prior knowledge, and efficient data incorporation.
Introduction:
Deep learning's effectiveness in AI with large-scale data.
Comparison between neural networks and Gaussian processes for sequential learning.
Challenges in continual learning (CL) addressed by GPs' function space representation.
Background:
Supervised learning setup with inputs and outputs.
Bayesian neural networks (BNNs) and posterior distribution calculations.
Laplace approximation for weight posterior estimation.
SFR: Sparse Function-space Representation:
Conversion of trained NNs into GPs using dual parameterization.
Sparsification using inducing points for computational efficiency.
Dual parameters calculation simplification compared to previous methods.
Experiments:
Supervised Learning:
Evaluation on UCI classification tasks and image datasets.
Comparison with GP subset method and Laplace approximation.
Sequential Learning:
Continual learning experiments showing SFR's effectiveness compared to other methods.
Incorporating New Data:
Fast incorporation of new data using SFR's dual updates compared to retraining from scratch.
Further Research Directions:
Investigate the impact of SFR's dual updates in downstream applications where retraining is costly or impractical.
Stats
この論文はICLR 2024で発表されました。
ニューラルネットワークを関数空間に変換するための新しい手法が導入されています。
Quotes
"Sequential learning paradigms pose challenges for gradient-based deep learning."
"We introduce a technique that converts neural networks from weight space to function space."