
The Neural Feature Ansatz (NFA), which observes a correlation between the Gram matrix of weights (NFM) and the average gradient outer product (AGOP) in trained neural networks, emerges from the alignment of weight matrices with pre-activation tangent kernel features driven by gradient descent.


coremsg

the-role-of-gradient-descent-in-feature-learning-aligning-weights-with-pre-activation-tangent-features-in-neural-networks


The Role of Gradient Descent in Feature Learning: Aligning Weights with Pre-Activation Tangent Features in Neural Networks


title_rewrite


This research paper demonstrates that gradient descent, when applied to two-layer neural networks with a carefully regularized objective, can learn useful features not only in the early stages of training but also in the later stages, leading to convergence to the ground-truth features.


feature-learning-in-two-layer-neural-networks-a-local-analysis-of-gradient-descent-with-regularization


Feature Learning in Two-Layer Neural Networks: A Local Analysis of Gradient Descent with Regularization
