Core Concepts
BNNs with general priors achieve optimal posterior concentration rates.
Abstract
The article discusses the importance of Bayesian Neural Networks (BNNs) in machine learning, focusing on their ability to combine deep neural networks and Bayesian techniques. It highlights the success of BNNs in various applications and the theoretical properties related to posterior concentrations. The lack of theoretical results for BNNs using Gaussian priors is addressed, introducing a new approximation theory for non-sparse DNNs with bounded parameters. The study shows that BNNs with non-sparse general priors can achieve near-minimax optimal posterior concentration rates to the true model.
The content is structured into sections discussing Bayesian approaches, properties of BNNs, approximation theories for DNNs, posterior concentration results for BNNs, and avoiding the curse of dimensionality by assuming hierarchical composition structure.
Stats
Surprisingly, there is no theoretical result about BNNs with i.i.d. standard Gaussian priors on the weights and biases.
Existing approximation theories of DNNs require weights to be either sparse or unbounded.
The ReLU activation function is extended to Leaky-ReLU activation function known for optimization merits.