Disentangling the Prediction Strategies of Neural Networks through Relevant Subspace Analysis
The core message of this article is to propose two novel analyses, Principal Relevant Component Analysis (PRCA) and Disentangled Relevant Subspace Analysis (DRSA), that can extract subspaces from the activations of a neural network that are maximally relevant to its prediction strategy. These disentangled subspaces enable more informative and structured explanations of the model's decision-making process.
Simultaneous Linear Connectivity of Neural Networks Modulo Permutation
Neural networks exhibit permutation symmetry, where reordering neurons in each layer does not change the underlying function they compute. This contributes to the non-convexity of the networks' loss landscapes. Recent work has argued that permutation symmetries are the only sources of non-convexity, meaning there are essentially no loss barriers between trained networks if they are permuted appropriately. This work refines these arguments into three distinct claims of increasing strength, and provides empirical evidence for the strongest claim.
Accurate Computation and Verification of Local Lipschitz Constant for ReLU-based Feedforward Neural Networks
This paper proposes a method to accurately compute the local Lipschitz constant of feedforward neural networks with ReLU activation functions, and derives a condition to verify the exactness of the computed upper bound.
Efficient Compositional Estimation of Lipschitz Constants for Deep Neural Networks
A compositional approach to efficiently estimate tight upper bounds on the Lipschitz constant of deep feedforward neural networks by decomposing the large matrix verification problem into smaller sub-problems that can be solved layer-by-layer.
Analyzing the Tradeoffs of Diagonal Fisher Information Matrix Estimators in Neural Networks
The Fisher information matrix characterizes the local geometry of the parameter space in neural networks. Due to its high computational cost, practitioners often use random estimators and evaluate only the diagonal entries. This work examines two such estimators, deriving bounds on their accuracy and sample complexity based on the variances associated with different parameter groups and the non-linearity of the network.
Topological Complexity of ReLU Neural Network Functions
The authors develop new local and global notions of topological complexity for fully-connected feedforward ReLU neural network functions, drawing on algebraic topology and a piecewise-linear version of Morse theory.
Analyzing the Complexity and Sensitivity of Neural Network Functions through the Boolean Mean Dimension
The Boolean Mean Dimension (BMD) can be used as a proxy for the sensitivity and complexity of neural network functions. The BMD exhibits a peak around the interpolation threshold, coinciding with the generalization error peak, and then decreases as the model becomes more overparameterized.
© 2024 by Linnk AI