Stabilizing Orthogonal Bases in Neural Networks Enables Efficient Low-Rank Training

The orthogonal component of a neural network's weights stabilizes early in the training process, enabling efficient low-rank training methods that maintain accuracy while significantly reducing the number of trainable parameters.