Log på
indsigt - Catapult Dynamics in SGD
Catapults in SGD: Auswirkungen auf Generalisierung durch Feature-Learning
Catapults in SGD führen zu besserer Generalisierung durch Feature-Learning.
Understanding Catapult Dynamics in Stochastic Gradient Descent
The author explains how spikes in training loss during SGD are caused by catapult dynamics in the top eigenspace of the tangent kernel. Furthermore, they demonstrate that these catapults lead to better generalization through increased alignment with the Average Gradient Outer Product (AGOP).
Produkter
© 2024 by Linnk AI