The paper introduces a new bi-level projection method that can efficiently enforce structured sparsity, particularly the ℓ1,∞ norm, in neural networks. The key idea is to split the projection into two simpler steps: first aggregating the columns using the q-norm, then projecting the aggregated vector onto the p-norm ball.
The authors show that this bi-level approach has a time complexity of O(nm) for a matrix in Rn×m, compared to O(nm log(nm)) for the best existing ℓ1,∞ projection algorithm. They also generalize the bi-level approach to a multi-level projection, which can achieve an exponential parallel speedup.
Experiments show that the bi-level ℓ1,∞ projection is 2.5 times faster than the state-of-the-art method while providing the same accuracy and better sparsity in neural network applications. The authors also demonstrate the application of their bi-level and multi-level projections to other structured sparsity norms like ℓ1,1 and ℓ1,2.
לשפה אחרת
מתוכן המקור
arxiv.org
תובנות מפתח מזוקקות מ:
by Guillaume Pe... ב- arxiv.org 05-06-2024
https://arxiv.org/pdf/2405.02086.pdfשאלות מעמיקות