핵심 개념
Implicit regularization through gradient flow minimizes nuclear norm of attention weights.
초록
The study explores how gradient flow implicitly regularizes the nuclear norm of attention weights in one-layer softmax models. It contrasts with prior results on Frobenius norm regularization, showing convergence to optimal solutions for binary classification tasks. The alignment property simplifies dynamics, ensuring global optimality and minimal loss convergence. Assumptions on data separability and initialization conditions drive the analysis, revealing insights into training dynamics of attention-based models.
통계
Under a separability assumption, gradient flow converges to minimize nuclear norm.
For diagonal key and query matrices, implicit regularization is described by an SVM problem.
Alignment property simplifies dynamics for general weight configurations.
Gradient flow implicitly regularizes combined attention weights towards low-rank structures.
인용구
"Gradient flow implicitly minimizes the nuclear norm of the combined attention weights."
"Alignment property ensures preservation of key and query matrix structure along the trajectory."