Unconstrained Feature Modeling Reveals Low-Dimensional Structures in Deep Neural Networks

The deep linear unconstrained feature model can theoretically demonstrate various low-dimensional structures observed in the weights, Hessians, gradients, and feature vectors of modern deep neural networks, which are caused by the emergence of deep neural collapse.